repeated builds are not deterministic #647
Comments
|
forgot to mention - I was able to notice these diffs on repeated builds of GoogleSans.designspace, |
|
In the past I have seen things like this arise when using collections that provide a non-deterministic iteration order (e.g. hashmaps) so that would be one thing I would look into first. I have done some work in the past in this regard (for instance #561) and it's very possible there are other places where we've missed this. |
|
I think here it's also to do with the fact that we split some tasks in parallel threads that may complete at different times and then we gather their result in some other collection where the order does matter |
|
yes that is totally possible as well. |
|
Good catch. Can we reproduce in a test with our test files? - that should make it easy to fix and keep fixed. |
|
I don't think our test files contain enough kerning pairs to trigger this.. |
|
Let's fix this after we land #644. We can add a test file with lots of kerns or a knob to help it trigger with fewer kerns if necessary. |
|
In IM @anthrotype noted "the GPOS vary in size between 106684 bytes and 4384362, an order of magnitude different.."; that's mildly alarming |
|
Reproduction: $ rm *.ttxl && for i in {0..9}; do rm -rf build/ && cargo run ../googlesans/source/GoogleSans/GoogleSans.designspace && ttx -l build/font.ttf > run$i.ttxl; done
$ grep -H GPOS run?.ttxl
run0.ttxl: GPOS 0xE96ECCD8 4384230 18876
run1.ttxl: GPOS 0x45C3186D 4384236 18880
run2.ttxl: GPOS 0xF17589DD 4384230 18876
run3.ttxl: GPOS 0xB16C1453 106684 15152
run4.ttxl: GPOS 0xA721CEE8 4384236 18876
run5.ttxl: GPOS 0x1E2BB2BF 4384230 18880
run6.ttxl: GPOS 0xB8849B8C 4384236 18876
run7.ttxl: GPOS 0xB16C1453 106684 15152
run8.ttxl: GPOS 0x2B12F05E 4384194 18876
run9.ttxl: GPOS 0x0E7DA5D9 4384188 18876 |
|
#647 (comment) repro is fixed, new repro: f=../googlesans/source/GoogleSans/GoogleSans.designspace; \
export SOURCE_DATE_EPOCH=$(date +%s); \
cargo build --release && rm -rf build/ \
&& target/release/fontc $f && cp build/font.ttf build/first.ttf \
&& target/release/fontc $f && cp build/font.ttf build/second.ttf \
&& diff -u <(fonttools ttx -o - -l build/first.ttf | tail -n +2) <(fonttools ttx -o - -l build/second.ttf | tail -n +2)
# GSUB and gvar report same length and offset but different contents
$ diff -u <(fonttools ttx -o - -t GSUB build/first.ttf | tail -n +2) <(fonttools ttx -o - -t GSUB build/font.ttf | tail -n +2)
# oh no, we're assigning inconsistent name ids, e.g.
- <UINameID value="263"/> <!-- Text Medium -->
+ <UINameID value="260"/> <!-- Medium --> |
|
The UINameID diff is fixed, table is now the same length, same offset, but binary different run to run. Steps to reproduce: f=../googlesans/source/GoogleSans/GoogleSans.designspace; \
export SOURCE_DATE_EPOCH=$(date +%s); \
cargo build --release && rm -rf build/ \
&& target/release/fontc $f && cp build/font.ttf build/first.ttf \
&& target/release/fontc $f && cp build/font.ttf build/second.ttf \
&& diff -u <(fonttools ttx -o - -l build/first.ttf | tail -n +2) <(fonttools ttx -o - -l build/second.ttf | tail -n +2)To dig further I have a fonttools hack. To view the code see fonttools/fonttools@main...rsheeter:fonttools:main. To use it: # Example assumes your currend directory is the root of fontc
# onetime setup
$ (cd .. && git clone [email protected]:rsheeter/fonttools.git fonttools_rs)
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -e ../fonttools_rs/
$ pip install absl-py
# Run the steps to reproduce (start of this comment). Should report GSUB mismatch, checksums differ, same length and offset
$ python ../fonttools_rs/Snippets/compare_table.py build/first.ttf build/second.ttf --noprint_eq
# should print a bunch of things with different offsets, abbreviated example follows
font["GSUB"].table.LookupList.Lookup[1383].SubTable[0].ExtSubTable.SubstLookupRecord[0].offset Value 86398 != 114938
font["GSUB"].table.LookupList.Lookup[1391].SubTable[1].ExtSubTable.SubstLookupRecord[0].offset Value 86838 != 98408
font["GSUB"].table.LookupList.Lookup[1395].SubTable[0].ExtSubTable.ChainSubRuleSet[0].ChainSubRule[2].SubstLookupRecord[0].offset Value 94840 != 113000
font["GSUB"].table.LookupList.Lookup[1395].SubTable[0].ExtSubTable.ChainSubRuleSet[0].ChainSubRule[3].SubstLookupRecord[0].offset Value 94854 != 113014
font["GSUB"].table.LookupList.Lookup[1395].SubTable[0].ExtSubTable.ChainSubRuleSet[1].ChainSubRule[0].SubstLookupRecord[0].offset Value 94784 != 112944
font["GSUB"].table.LookupList.Lookup[1395].SubTable[0].ExtSubTable.ChainSubRuleSet[1].ChainSubRule[1].SubstLookupRecord[0].offset Value 94798 != 112958
font["GSUB"].table.LookupList.Lookup[1408].SubTable[0].ExtSubTable.ChainSubClassSet[1].ChainSubClassRule[22].SubstLookupRecord[0].offset Value 79246 != 87712 |
|
Passing to @cmyr to dig into packing |
|
googlefonts/fontations#738 fixed indeterminism in GSUB table, however the gvar table is still changing each time fontc is run on GS (same length, different bytes/checksum) see #656 |
|
to clarify my previous comment, only the gvar table bytes are different, its high-level content as dumped with ttx is the same at each re-run. |
Meant to reproduce the issue discussed at googlefonts/fontc#647 If we are lucky enough the CI should fail, otherwise flip the coin again and you'll see...
|
looks like GDEF and GPOS still are not deterministic... :( https://github.com/googlefonts/fontc/actions/runs/7501084544/job/20421031213?pr=674#step:12:32 they appeared to have changed length (and thus checksum) between the first and second build run |
|
oh dear.. I can't reproduce locally. No idea what is going on :( |
|
baging my head, still can't repro, but I can say that I just re-run the CI jobs that had previously passed on |
|
This is the shell script I'm using to attempt to reproduce locally what the CI is doing, I tried on both my glinux workstation and on my macbook pro, both are using the latest rust toolchain 1.75.0, I did cargo update, even cargo clean, I still can't reproduce, this script always ends without error and the files are identical. #!/bin/bash
# clean up
rm -rf first-roman.ttf second-roman.ttf first-italic.ttf second-italic.ttf
rm -rf build
echo "Installing fontc"
(cd fontc && cargo install --path .)
which fontc
echo "Building the first roman"
time fontc ../googlesans/source/GoogleSans/GoogleSans.designspace
mv build/font.ttf ./first-roman.ttf
echo "Building the first italic"
time fontc ../googlesans/source/GoogleSans/GoogleSans-Italic.designspace
mv build/font.ttf ./first-italic.ttf
echo "Building the second roman"
time fontc ../googlesans/source/GoogleSans/GoogleSans.designspace
mv build/font.ttf ./second-roman.ttf
echo "Building the second italic"
time fontc ../googlesans/source/GoogleSans/GoogleSans-Italic.designspace
mv build/font.ttf ./second-italic.ttf
echo "diff {first,second}-roman.ttf"
diff -u <(ttx -l first-roman.ttf | tail -n +2) <(ttx -l second-roman.ttf | tail -n +2)
cmp first-roman.ttf second-roman.ttf
if [ $? -ne 0 ]; then
echo "error: non repeatable build :("
fi
echo "diff {first,second}-italic.ttf"
diff -u <(ttx -l first-italic.ttf | tail -n +2) <(ttx -l second-italic.ttf | tail -n +2)
cmp first-italic.ttf second-italic.ttf
if [ $? -ne 0 ]; then
echo "error: non repeatable build :("
exit 1
else
echo "success! :)"
exit 0
fi |
|
i'm going crazy chasing this heisenbug 🤯 In #676 I added a shell script (so that I can try to run the same commands locally and on CI) that builds a font twice and diff the ttx -l and cmp the binary fonts but now I can't seem to reproduce the failure any more on the CI even (locally, I never managed to repro). |
|
and to add insult to injury, my repeatable-builds.sh script is now failing with Oswald, complains the gvar bytes changed! But only sometimes.. 😢 |
|
I can reproduce GS build instability locally as follows: rm -f *.ttxl threads*.svg && for i in {0..4}; do rm -rf build/ && cargo run ../googlesans/source/GoogleSans/GoogleSans.designspace --emit-timing && cp build/threads.svg ./threads_$i.svg && ttx -l build/font.ttf > run$i.ttxl && grep GPOS *.ttxl; doneLooking at the threads_#.svg for a failure (small GPOS) shows we are running gather BE kerning before we generate it: |
|
I managed to isolate the gvar indeterminism, it's the shared tuples array that may end up being sorted differently each time because we are using a HashMap to count how many times a given tuple is shared among glyph tuple variations, and then use sort_unstable_by_key to take the max, but the sorting method does not preserve the order of equal elements (and HashMap also is inherently unsorted so iteration order isn't guaranteed either). If I use IndexMap::sort_by_key the order is stable and builds are reproducible again. I'll send a PR shortly. |
failing test meant to reproduce googlefonts/fontc#647 (comment)
ok the head.modified changes every time (and checksums would also change accoridngly), but I would not expect other things to differ when I compare ttx dumps of font files that were freshly built from the same source file using the same copy of fontc.
It turns out there are some differences in things like GDEF and GPOS (as noted in #644 (comment), this one possible regression introduced with #642), as well as in GSUB (in the FeatureParamsStylisticSet nameIDs).
Also when doing
ttx -lto show the table directory, the gvar table seems to have different checksums each time round, even though the length is the same.. But diffing its ttx repesentations gives no differences, so it might be the bytes themselves are serialized differently each time, for some reasons.The text was updated successfully, but these errors were encountered: