Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix artificial language matching #253

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

bep
Copy link
Contributor

@bep bep commented Apr 24, 2021

This works around what seems to be an upstream by implementing a simplified tag matcher for artificual languages.

Fixes #252

I have run the relevant Hugo benchmarks (which I'm more familiar with) and, not surprisingly, this is faster in the "artificial case", other than that it should be about the same.

name                                              old time/op    new time/op    delta
I18nTranslate/all-present-16                         930ns ± 1%     926ns ± 0%     ~     (p=0.343 n=4+4)
I18nTranslate/present-in-default-16                 1.63µs ± 1%    1.62µs ± 1%     ~     (p=0.143 n=4+4)
I18nTranslate/present-in-current-16                  933ns ± 0%     922ns ± 1%   -1.14%  (p=0.029 n=4+4)
I18nTranslate/missing-16                            1.50µs ± 0%    1.49µs ± 0%     ~     (p=0.114 n=4+4)
I18nTranslate/file-missing-16                       3.05µs ± 0%    3.07µs ± 0%   +0.67%  (p=0.029 n=4+4)
I18nTranslate/context-provided-16                   2.36µs ± 1%    2.33µs ± 0%   -1.24%  (p=0.029 n=4+4)
I18nTranslate/readingTime-one-16                     671ns ± 1%     665ns ± 1%     ~     (p=0.314 n=4+4)
I18nTranslate/readingTime-many-dot-16               1.45µs ± 1%    1.41µs ± 0%   -3.21%  (p=0.029 n=4+4)
I18nTranslate/readingTime-many-16                   2.76µs ± 1%    2.86µs ± 0%   +3.85%  (p=0.029 n=4+4)
I18nTranslate/readingTime-map-one-16                 723ns ± 1%     725ns ± 0%     ~     (p=0.486 n=4+4)
I18nTranslate/readingTime-string-one-16              712ns ± 2%     702ns ± 1%     ~     (p=0.114 n=4+4)
I18nTranslate/readingTime-map-many-16               1.74µs ± 0%    1.69µs ± 2%   -2.83%  (p=0.029 n=4+4)
I18nTranslate/argument-float-16                     1.44µs ± 0%    1.39µs ± 1%   -3.54%  (p=0.029 n=4+4)
I18nTranslate/same-id-and-translation-16             916ns ± 2%     915ns ± 1%     ~     (p=0.886 n=4+4)
I18nTranslate/same-id-and-translation-default-16    1.61µs ± 0%    1.61µs ± 0%     ~     (p=0.114 n=4+4)
I18nTranslate/unknown-language-code-16              3.85µs ± 0%    3.18µs ± 1%  -17.31%  (p=0.029 n=4+4)
I18nTranslate/known-language-missing-plural-16      1.57µs ± 2%    0.89µs ± 3%  -43.12%  (p=0.029 n=4+4)
I18nTranslate/dotted-bare-key-16                     903ns ± 1%     909ns ± 1%     ~     (p=0.343 n=4+4)
I18nTranslate/lang-with-hyphen-16                    826ns ± 1%     827ns ± 0%     ~     (p=1.000 n=4+4)

name                                              old alloc/op   new alloc/op   delta
I18nTranslate/all-present-16                          384B ± 0%      384B ± 0%     ~     (all equal)
I18nTranslate/present-in-default-16                   496B ± 0%      496B ± 0%     ~     (all equal)
I18nTranslate/present-in-current-16                   384B ± 0%      384B ± 0%     ~     (all equal)
I18nTranslate/missing-16                              496B ± 0%      496B ± 0%     ~     (all equal)
I18nTranslate/file-missing-16                         672B ± 0%      672B ± 0%     ~     (all equal)
I18nTranslate/context-provided-16                     360B ± 0%      360B ± 0%     ~     (all equal)
I18nTranslate/readingTime-one-16                     48.0B ± 0%     48.0B ± 0%     ~     (all equal)
I18nTranslate/readingTime-many-dot-16                 232B ± 0%      232B ± 0%     ~     (all equal)
I18nTranslate/readingTime-many-16                     416B ± 0%      416B ± 0%     ~     (all equal)
I18nTranslate/readingTime-map-one-16                 48.0B ± 0%     48.0B ± 0%     ~     (all equal)
I18nTranslate/readingTime-string-one-16              48.0B ± 0%     48.0B ± 0%     ~     (all equal)
I18nTranslate/readingTime-map-many-16                 264B ± 0%      264B ± 0%     ~     (all equal)
I18nTranslate/argument-float-16                       224B ± 0%      224B ± 0%     ~     (all equal)
I18nTranslate/same-id-and-translation-16              384B ± 0%      384B ± 0%     ~     (all equal)
I18nTranslate/same-id-and-translation-default-16      496B ± 0%      496B ± 0%     ~     (all equal)
I18nTranslate/unknown-language-code-16                512B ± 0%      440B ± 0%  -14.06%  (p=0.029 n=4+4)
I18nTranslate/known-language-missing-plural-16        136B ± 0%       72B ± 0%  -47.06%  (p=0.029 n=4+4)
I18nTranslate/dotted-bare-key-16                      384B ± 0%      384B ± 0%     ~     (all equal)
I18nTranslate/lang-with-hyphen-16                    48.0B ± 0%     48.0B ± 0%     ~     (all equal)

name                                              old allocs/op  new allocs/op  delta
I18nTranslate/all-present-16                          3.00 ± 0%      3.00 ± 0%     ~     (all equal)
I18nTranslate/present-in-default-16                   8.00 ± 0%      8.00 ± 0%     ~     (all equal)
I18nTranslate/present-in-current-16                   3.00 ± 0%      3.00 ± 0%     ~     (all equal)
I18nTranslate/missing-16                              8.00 ± 0%      8.00 ± 0%     ~     (all equal)
I18nTranslate/file-missing-16                         15.0 ± 0%      15.0 ± 0%     ~     (all equal)
I18nTranslate/context-provided-16                     9.00 ± 0%      9.00 ± 0%     ~     (all equal)
I18nTranslate/readingTime-one-16                      1.00 ± 0%      1.00 ± 0%     ~     (all equal)
I18nTranslate/readingTime-many-dot-16                 5.00 ± 0%      5.00 ± 0%     ~     (all equal)
I18nTranslate/readingTime-many-16                     12.0 ± 0%      12.0 ± 0%     ~     (all equal)
I18nTranslate/readingTime-map-one-16                  1.00 ± 0%      1.00 ± 0%     ~     (all equal)
I18nTranslate/readingTime-string-one-16               1.00 ± 0%      1.00 ± 0%     ~     (all equal)
I18nTranslate/readingTime-map-many-16                 7.00 ± 0%      7.00 ± 0%     ~     (all equal)
I18nTranslate/argument-float-16                       5.00 ± 0%      5.00 ± 0%     ~     (all equal)
I18nTranslate/same-id-and-translation-16              3.00 ± 0%      3.00 ± 0%     ~     (all equal)
I18nTranslate/same-id-and-translation-default-16      8.00 ± 0%      8.00 ± 0%     ~     (all equal)
I18nTranslate/unknown-language-code-16                17.0 ± 0%      13.0 ± 0%  -23.53%  (p=0.029 n=4+4)
I18nTranslate/known-language-missing-plural-16        6.00 ± 0%      2.00 ± 0%  -66.67%  (p=0.029 n=4+4)
I18nTranslate/dotted-bare-key-16                      3.00 ± 0%      3.00 ± 0%     ~     (all equal)
I18nTranslate/lang-with-hyphen-16                     1.00 ± 0%      1.00 ± 0%     ~     (all equal)

@codecov
Copy link

codecov bot commented Apr 24, 2021

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.91%. Comparing base (482713a) to head (cba43ec).
Report is 54 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #253      +/-   ##
==========================================
+ Coverage   82.68%   82.91%   +0.23%     
==========================================
  Files          14       14              
  Lines        1178     1194      +16     
==========================================
+ Hits          974      990      +16     
  Misses        141      141              
  Partials       63       63              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bep bep force-pushed the art-lang branch 4 times, most recently from 0a6b4a9 to 5429ac2 Compare April 25, 2021 09:17
base, _ := candidate.Base()

if base != artTagBase {
break
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this will actually solve the problem your upstream users are facing? This logic here bails as soon as it sees a non-artificial language, so this workaround will only work if the artificial language is first in the list. Is that always the case with Hugo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hugo creates one Localizer per language, so there will always be only 1 language in that list. But you are right. I added that break as a premature optimization, I will fix it. Thanks.

This works around what seems to be an upstream by implementing a simplified tag matcher for artificual languages.

Fixes nicksnyder#252
@bep
Copy link
Contributor Author

bep commented Apr 26, 2021

I forced pushed an updated version where I "continue" instead of break. Also added another language to the test.

@bep
Copy link
Contributor Author

bep commented Apr 27, 2021

I would appreciate a merge of this sooner rather than later, as I need to get a fix out for this.

return language.NewMatcher(tags)
}

return matcher{
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hugo creates one Localizer per language

Oh interesting. Can we limit the scope of this workaround even further? For example, the new matcher should only activate when ALL the tags are artificial?

This is fundamentally an upstream issue so I don’t want to add a lot of non-intuitive complexity to go-i18n itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the new matcher should only activate when ALL the tags are artificial?

No, the tags set here is all the tags in the bundle. There will be a mix of real and artificial languages.

This is fundamentally an upstream issue so I don’t want to add a lot of non-intuitive complexity to go-i18n itself.

I'm just talking for myself here, but to me this is my issue; I had an extremely naive test with only 1 artificial language -- so with that (my) upgrade to the v2 package of go-i18n broke many Hugo sites in the wild, which is why I have said in another thread that I'm releasing a patch for this this week, one way or another.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I thought this patch was uncontroversial -- I'll try to find another temporary workaround. Thanks.

@flyn-org
Copy link

flyn-org commented May 5, 2022

Any hope for progress here? I package Hugo for Fedora, and the use of https://github.com/gohugoio/go-i18n rather than the package here makes that more difficult. We don't have a Fedora -devel package for Hugo's fork, whereas we do have a -devel package for the code here. I am hesitant to package the fork because it bears the warning "This is a short-lived fork to fix a critical upstream issue, do not use."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fails to resolve with multiple artificial languages
3 participants