Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix artificial language matching #253

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 51 additions & 2 deletions v2/i18n/bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,58 @@ type Bundle struct {
matcher language.Matcher
}

// The matcher in x/text/language does not handle artificial languages,
// see https://github.com/golang/go/issues/45749
// This is a simplified matcher that delegates to the x/text/language matcher for
// the harder cases.
type matcher struct {
tags []language.Tag
defaultMatcher language.Matcher
}

func newMatcher(tags []language.Tag) language.Matcher {
var hasArt bool
for _, tag := range tags {
base, _ := tag.Base()
hasArt = base == artTagBase
if hasArt {
break
}
}

if !hasArt {
return language.NewMatcher(tags)
}

return matcher{
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hugo creates one Localizer per language

Oh interesting. Can we limit the scope of this workaround even further? For example, the new matcher should only activate when ALL the tags are artificial?

This is fundamentally an upstream issue so I don’t want to add a lot of non-intuitive complexity to go-i18n itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the new matcher should only activate when ALL the tags are artificial?

No, the tags set here is all the tags in the bundle. There will be a mix of real and artificial languages.

This is fundamentally an upstream issue so I don’t want to add a lot of non-intuitive complexity to go-i18n itself.

I'm just talking for myself here, but to me this is my issue; I had an extremely naive test with only 1 artificial language -- so with that (my) upgrade to the v2 package of go-i18n broke many Hugo sites in the wild, which is why I have said in another thread that I'm releasing a patch for this this week, one way or another.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I thought this patch was uncontroversial -- I'll try to find another temporary workaround. Thanks.

tags: tags,
defaultMatcher: language.NewMatcher(tags),
}
}

func (m matcher) Match(t ...language.Tag) (language.Tag, int, language.Confidence) {
for _, candidate := range t {
base, _ := candidate.Base()
if base != artTagBase {
continue
}

for i, tag := range m.tags {
if tag == candidate {
return candidate, i, language.Exact
}
}
}

return m.defaultMatcher.Match(t...)
}

// artTag is the language tag used for artificial languages
// https://en.wikipedia.org/wiki/Codes_for_constructed_languages
var artTag = language.MustParse("art")
var (
artTag = language.MustParse("art")
artTagBase, _ = artTag.Base()
)

// NewBundle returns a bundle with a default language and a default set of plural rules.
func NewBundle(defaultLanguage language.Tag) *Bundle {
Expand Down Expand Up @@ -126,7 +175,7 @@ func (b *Bundle) addTag(tag language.Tag) {
}
}
b.tags = append(b.tags, tag)
b.matcher = language.NewMatcher(b.tags)
b.matcher = newMatcher(b.tags)
}

// LanguageTags returns the list of language tags
Expand Down
25 changes: 25 additions & 0 deletions v2/i18n/bundle_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,31 @@ hello = "`+expected+`"
}
}

func TestPseudoLanguages(t *testing.T) {
bundle := NewBundle(language.English)
bundle.RegisterUnmarshalFunc("toml", toml.Unmarshal)
expected := "a2"
bundle.MustParseMessageFileBytes([]byte(`
hello = "a1"
`), "art-x-a1.toml")
bundle.MustParseMessageFileBytes([]byte(`
hello = "a2"
`), "art-x-a2.toml")
bundle.MustParseMessageFileBytes([]byte(`
hello = "a3"
`), "art-x-a3.toml")

{
localized, err := NewLocalizer(bundle, "art-x-a2").Localize(&LocalizeConfig{MessageID: "hello"})
if err != nil {
t.Fatal(err)
}
if localized != expected {
t.Fatalf("expected %q\ngot %q", expected, localized)
}
}
}

func TestJSON(t *testing.T) {
bundle := NewBundle(language.English)
bundle.MustParseMessageFileBytes([]byte(`{
Expand Down