Move to nix buildstack (and nix develop produced docker image used under CircleCI) #1661
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tlaurion
force-pushed
the
wip-nix-for-build
branch
2 times, most recently
from
4f0aa80
to
c06e524
Compare
Noticed a couple of dupes (gz and rom) so I figured I should sort it now to hopefully avoid another one in the future. Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Remove hard coded paths from shebangs and other references because they do not play well in nix-land. Either use /usr/bin/env to do runtime PATH based lookup or avoid absolute paths so PATH look up happens instead. Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Just some minor clean ups like fixing whitespace and sorting things. I added (bash)/removed (libusb) white space in order to look like the other modules. I sorted the --enable/--disable/--with blocks so that common stuff looked similar which should aid in comparing modules. I also removed a couple of duplicate config options (--disable-fallback-curses & --disable-regex). Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
More pending work needed to fix fragility of buildsystem and fix nix build issues as well like: https://app.circleci.com/pipelines/github/mmlb/osresearch-heads/11/workflows/32cc883c-5074-4f28-94b8-a83a2ec44414/jobs/252 https://app.circleci.com/pipelines/github/mmlb/osresearch-heads/11/workflows/32cc883c-5074-4f28-94b8-a83a2ec44414/jobs/221 https://app.circleci.com/pipelines/github/tlaurion/heads/1781/workflows/ee402ead-6739-4549-88ae-105b695fb3cd https://app.circleci.com/pipelines/github/tlaurion/heads/1783/workflows/2b35826c-aff4-4f48-8809-4e66259f9aa4/jobs/25877/parallel-runs/0/steps/0-103 Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Thanks to @JonathonHall-Purism, that pointed to me that sysroot was neglected in tpm2-tools configure step. Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
… args Thanks to @JonathonHall-Purism, that pointed to me that sysroot was neglected in tpm2-tools configure step. I wonder why this is not respected if not forced with --with-sysroot and TSS2_ESYS_3_0_LIBS="-ltss2-esys -L$(INSTALL)/lib"? Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
These are generated during the build process so should be ignored. Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Got rid of long lines in favor of more lines for readability. Cleaned up some comments/typos and unnecessary cruft*. Finally ran prettier on the file for its automatic formatting, including whitespace clean ups. cruft: - && when already set -e - run commands with trailing \ - deleted commented out "OLD STUFF" - sorted listy looking things because unsorted stuff bothers me :) (I held back on sorting the board build definitions though, thats probably too much). Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Gives the exact same output:
```
docker run --rm -ti debian:11 bash -c '
mkdir -p build/subdir1/ build/subdir2
echo "subdir1 error" >build/subdir1/fail.log
echo "subdir2 error" >build/subdir2/fail.log
find build -type f -name "*.log" -exec tail -n +1 "{}" +
'
==> build/subdir1/fail.log <==
subdir1 error
==> build/subdir2/fail.log <==
subdir2 error
```
Signed-off-by: Manuel Mendez <[email protected]>
Signed-off-by: Thierry Laurion <[email protected]>
Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
Until nix PR is merged to not interfere with master/other pr caches Signed-off-by: Manuel Mendez <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
…maximized Signed-off-by: Thierry Laurion <[email protected]>
…now passed from golbal Makefile sysroot (TODO: generalize) Signed-off-by: Thierry Laurion <[email protected]>
…es to not delete already downloaded packages to economize bandwidth Signed-off-by: Thierry Laurion <[email protected]>
/nix/store/5lr5n3qa4day8l1ivbwlcby2nknczqkq-bash-5.2p26/bin/bash ./libtool --tag=CC --mode=link /home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/home/user/heads=heads -gno-record-gcc-switches -D__MUSL__ --sysroot /home/user/heads/install/x86 -isystem /home/user/heads/install/x86/include -L/home/user/heads/install/x86/lib -I./tools -I./lib -Wall -Wextra -Wformat -Wformat-security -Wstack-protector -fstack-protector-all -Wstrict-overflow=5 -O2 -fPIC -fPIE -D_GNU_SOURCE -std=gnu99 -Wstringop-overflow=4 -Wstringop-truncation -Wduplicated-branches -Wduplicated-cond -Wbool-compare -fdata-sections -ffunction-sections -I/home/user/heads/install/x86/include -I/home/user/heads/install/x86//include -I/home/user/heads/install/x86//include/tss2 -I/home/user/heads/install/x86/nix/store/yg75achq89wgqn2fi3gglgsd77kjpi03-openssl-3.0.13-dev/include -I/home/user/heads/install/x86//include -I/home/user/heads/install/x86//include/tss2 -I/home/user/heads/install/x86//include -I/home/user/heads/install/x86//include/tss2 -I/home/user/heads/install/x86//include -I/home/user/heads/install/x86//include/tss2 -DTPM2_TOOLS_MAX="101" -fdebug-prefix-map=/home/user/heads/install/x86=. -shared -pie -Wl,-z,relro -Wl,-z,now -Wl,--gc-sections -o tools/tpm2 tools/tpm2-tpm2_tool.o tools/misc/tpm2-tpm2_certifyX509certutil.o tools/misc/tpm2-tpm2_checkquote.o tools/misc/tpm2-tpm2_encodeobject.o tools/misc/tpm2-tpm2_eventlog.o tools/misc/tpm2-tpm2_print.o tools/misc/tpm2-tpm2_rc_decode.o tools/misc/tpm2-tpm2_tr_encode.o tools/tpm2-tpm2_activatecredential.o tools/tpm2-tpm2_certify.o tools/tpm2-tpm2_changeauth.o tools/tpm2-tpm2_changeeps.o tools/tpm2-tpm2_changepps.o tools/tpm2-tpm2_clear.o tools/tpm2-tpm2_clearcontrol.o tools/tpm2-tpm2_clockrateadjust.o tools/tpm2-tpm2_create.o tools/tpm2-tpm2_createak.o tools/tpm2-tpm2_createek.o tools/tpm2-tpm2_createpolicy.o tools/tpm2-tpm2_setprimarypolicy.o tools/tpm2-tpm2_createprimary.o tools/tpm2-tpm2_dictionarylockout.o tools/tpm2-tpm2_duplicate.o tools/tpm2-tpm2_getcap.o tools/tpm2-tpm2_gettestresult.o tools/tpm2-tpm2_encryptdecrypt.o tools/tpm2-tpm2_evictcontrol.o tools/tpm2-tpm2_flushcontext.o tools/tpm2-tpm2_getrandom.o tools/tpm2-tpm2_gettime.o tools/tpm2-tpm2_hash.o tools/tpm2-tpm2_hierarchycontrol.o tools/tpm2-tpm2_hmac.o tools/tpm2-tpm2_import.o tools/tpm2-tpm2_incrementalselftest.o tools/tpm2-tpm2_load.o tools/tpm2-tpm2_loadexternal.o tools/tpm2-tpm2_makecredential.o tools/tpm2-tpm2_nvdefine.o tools/tpm2-tpm2_nvextend.o tools/tpm2-tpm2_nvincrement.o tools/tpm2-tpm2_nvreadpublic.o tools/tpm2-tpm2_nvread.o tools/tpm2-tpm2_nvreadlock.o tools/tpm2-tpm2_nvundefine.o tools/tpm2-tpm2_nvwrite.o tools/tpm2-tpm2_nvwritelock.o tools/tpm2-tpm2_nvsetbits.o tools/tpm2-tpm2_pcrallocate.o tools/tpm2-tpm2_pcrevent.o tools/tpm2-tpm2_pcrextend.o tools/tpm2-tpm2_pcrread.o tools/tpm2-tpm2_pcrreset.o tools/tpm2-tpm2_policypcr.o tools/tpm2-tpm2_policyauthorize.o tools/tpm2-tpm2_policyauthorizenv.o tools/tpm2-tpm2_policynv.o tools/tpm2-tpm2_policycountertimer.o tools/tpm2-tpm2_policyor.o tools/tpm2-tpm2_policynamehash.o tools/tpm2-tpm2_policytemplate.o tools/tpm2-tpm2_policycphash.o tools/tpm2-tpm2_policypassword.o tools/tpm2-tpm2_policysigned.o tools/tpm2-tpm2_policyticket.o tools/tpm2-tpm2_policyauthvalue.o tools/tpm2-tpm2_policysecret.o tools/tpm2-tpm2_policyrestart.o tools/tpm2-tpm2_policycommandcode.o tools/tpm2-tpm2_policynvwritten.o tools/tpm2-tpm2_policyduplicationselect.o tools/tpm2-tpm2_policylocality.o tools/tpm2-tpm2_quote.o tools/tpm2-tpm2_readclock.o tools/tpm2-tpm2_readpublic.o tools/tpm2-tpm2_rsadecrypt.o tools/tpm2-tpm2_rsaencrypt.o tools/tpm2-tpm2_send.o tools/tpm2-tpm2_selftest.o tools/tpm2-tpm2_setclock.o tools/tpm2-tpm2_shutdown.o tools/tpm2-tpm2_sign.o tools/tpm2-tpm2_certifycreation.o tools/tpm2-tpm2_nvcertify.o tools/tpm2-tpm2_startauthsession.o tools/tpm2-tpm2_startup.o tools/tpm2-tpm2_stirrandom.o tools/tpm2-tpm2_testparms.o tools/tpm2-tpm2_unseal.o tools/tpm2-tpm2_verifysignature.o tools/tpm2-tpm2_setcommandauditstatus.o tools/tpm2-tpm2_getcommandauditdigest.o tools/tpm2-tpm2_getsessionauditdigest.o tools/tpm2-tpm2_geteccparameters.o tools/tpm2-tpm2_ecephemeral.o tools/tpm2-tpm2_commit.o tools/tpm2-tpm2_ecdhkeygen.o tools/tpm2-tpm2_ecdhzgen.o tools/tpm2-tpm2_zgen2phase.o tools/tpm2-tpm2_sessionconfig.o tools/tpm2-tpm2_getpolicydigest.o lib/libcommon.a -ltss2-esys -L/home/user/heads/install/x86/lib -L/home/user/heads/install/x86//lib -ltss2-mu -L/home/user/heads/install/x86/nix/store/7nmrrad8skxr47f9hfl3xc0pfqmwq51b-openssl-3.0.13/lib -lcrypto -L/home/user/heads/install/x86//lib -ltss2-tctildr -L/home/user/heads/install/x86//lib -ltss2-rc -L/home/user/heads/install/x86//lib -ltss2-sys libtool: error: cannot find the library '//lib/libtss2-sys.la' or unhandled argument '//lib/libtss2-sys.la' make[1]: *** [Makefile:2478: tools/tpm2] Error 1 make[1]: Leaving directory '/home/user/heads/build/x86/tpm2-tools-5.6' make: *** [Makefile:521: /home/user/heads/build/x86/tpm2-tools-5.6/.build] Error 1 Signed-off-by: Thierry Laurion <[email protected]>
…test CPUS=8 to maximize loadavg on CircleCI with 4 CPUs & 8GB ram See first lines of output of any make command. Change aimed to be respectful of CI resource (8GB ram 4CPUs) With CPUS=8 AVAILABLE_MEM_GB=4, CircleCI outputs: !!!!!! BUILD SYSTEM INFO !!!!!! System CPUS: 36 System Available Memory: 4 GB System Load Average: 12.99 ---------------------------------------------------------------------- Used **CPUS**: 8 Used **LOADAVG**: 8 Used **AVAILABLE_MEM_GB**: 4 GB ---------------------------------------------------------------------- **MAKE_JOBS**: -j8 --max-load 8 Variables available for override (use 'make VAR_NAME=value'): **CPUS** (default: number of processors, e.g., 'make CPUS=4') **LOADAVG** (default: same as CPUS, e.g., 'make LOADAVG=4') **AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4') **MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2') ---------------------------------------------------------------------- Let's try without any limitation... Signed-off-by: Thierry Laurion <[email protected]>
Signed-off-by: Thierry Laurion <[email protected]>
…at were missing it to propogate build optimizations per module, while still impossible to call make -j 12 on main make call Signed-off-by: Thierry Laurion <[email protected]>
…ch_rebuild_what_changed: reextract packages, repatch sources (might fail, easy to fix) and rebuild only what changed (not a lot) if patch fails to apply, its because patch file creates a file and doesn't expect it to exist. just call rm on the file reported to exist, and relaunch build. Deletes ./install/*/* and permits to rebuild all dependencies in order, just based on freshly extracted and patched code. Bonus, this saves your SDD from unneeded wear and rebuilds faster then all other Mafile helpers. That's my favorite. Signed-off-by: Thierry Laurion <[email protected]>
…t loosing traces of reproduction notes Signed-off-by: Thierry Laurion <[email protected]>
… creation. Cleaner NIX_REPRO_NOTES Signed-off-by: Thierry Laurion <[email protected]>
…elop) which fails at tpm2-tss - switch cache to nix-docker-heads to not interfere with nixos develop layer on same PR - remove nix develop calls; replace by direct script calls and make calls - make sure save/restore/root is ~/heads Signed-off-by: Thierry Laurion <[email protected]> Signed-off-by: Thierry Laurion <[email protected]>
….com/pipelines/github/tlaurion/heads/2500/workflows/23674215-8b22-4852-adf4-2a6df9e44353/jobs/45080?invite=true#step-102-16530_106 zlib-dev not found on coreboot buildstack buildstep... Signed-off-by: Thierry Laurion <[email protected]>
… to zlib-ng Signed-off-by: Thierry Laurion <[email protected]>
…circleci.com/pipelines/github/tlaurion/heads/2500/workflows/23674215-8b22-4852-adf4-2a6df9e44353/jobs/45080?invite=true#step-102-16530_106 zlib-dev not found on coreboot buildstack buildstep..." This reverts commit d6c4996. Signed-off-by: Thierry Laurion <[email protected]>
2 tasks
JonathonHall-Purism
approved these changes
May 10, 2024
tlaurion
force-pushed
the
wip-nix-for-build
branch
3 times, most recently
from
1defde9
to
ebe31a7
Compare
…instructions Signed-off-by: Thierry Laurion <[email protected]>
tlaurion
force-pushed
the
wip-nix-for-build
branch
from
ebe31a7
to
ecbfdbc
Compare
|
Some last forced pushes to have the README.md kosher prior of merging! |
|
And now, let's enjoy reproducible buildstack and roms with less maintenance burden. Aho. |
This was referenced May 10, 2024
tlaurion
added a commit
to tlaurion/heads-wiki
that referenced
this pull request
May 12, 2024
… was merged (linuxboot/heads#1661) Signed-off-by: Thierry Laurion <[email protected]>
This was referenced May 12, 2024
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
May 17, 2024
…rd on master (Opt: Authenticated Heads) - Revert gnupg toolstack version bump to prior of linuxboot#1661 merge (2.4.2 -> 2.4.0). Version bump not needed for reproducibility. - Investigation and upstream discussions will take their time resolving invalid time issue introduced by between 2.4.0 and latest gnupg, fix regression first under master) - oem-factory-reset - Adding DO_WITH_DEBUG to oem-factory-reset for all its gpg calls. If failing in debug mode, /tmp/debug.txt contains calls and errors - Wipe keyrings only (*.gpg, *.kbx) not conf files under gpg homedir (keep initrd/.gnupg/*.conf) - flake.nix - switch build derivative from qemu and qemu_kvm to qemu_full to have qemu-img tool which was missing to run qemu boards (v0.1.8 docker) - add gnupg so that qemu boards can call inject_gpg to inject public key in absence of flashrom+pflash support for internal flashing - flake.lock: Updated nix pinned package list under flake.lock with 'nix flake update' so qemu_full builds - README.md: have consistent docker testing + release (push) notes - .circleci/config.yml: depend on docker v0.1.8 (qemu_full built with canokey-qemu lib support, diffoscopeMinimal and gnupg for proper qemu testing) TODO: - some fd2 instead of fd1?! - oem-factory-resest has whiptail_or_die which sets whiptail box to HEIGHT 0. This doesn't show a scrolling window on gpg errors which is problematic with fbwhiptail, not whiptail Signed-off-by: Thierry Laurion <[email protected]>
tlaurion
added a commit
to tlaurion/heads
that referenced
this pull request
May 17, 2024
…h from linuxboot#1661 (less and less required but still some). Cannot remove 5.10.5 because kgpe-d16 uses it. Signed-off-by: Thierry Laurion <[email protected]>
mdrobnak
pushed a commit
to mdrobnak/heads
that referenced
this pull request
May 28, 2024
…rd on master (Opt: Authenticated Heads) - Revert gnupg toolstack version bump to prior of linuxboot#1661 merge (2.4.2 -> 2.4.0). Version bump not needed for reproducibility. - Investigation and upstream discussions will take their time resolving invalid time issue introduced by between 2.4.0 and latest gnupg, fix regression first under master) - oem-factory-reset - Adding DO_WITH_DEBUG to oem-factory-reset for all its gpg calls. If failing in debug mode, /tmp/debug.txt contains calls and errors - Wipe keyrings only (*.gpg, *.kbx) not conf files under gpg homedir (keep initrd/.gnupg/*.conf) - flake.nix - switch build derivative from qemu and qemu_kvm to qemu_full to have qemu-img tool which was missing to run qemu boards (v0.1.8 docker) - add gnupg so that qemu boards can call inject_gpg to inject public key in absence of flashrom+pflash support for internal flashing - flake.lock: Updated nix pinned package list under flake.lock with 'nix flake update' so qemu_full builds - README.md: have consistent docker testing + release (push) notes - .circleci/config.yml: depend on docker v0.1.8 (qemu_full built with canokey-qemu lib support, diffoscopeMinimal and gnupg for proper qemu testing) TODO: - some fd2 instead of fd1?! - oem-factory-resest has whiptail_or_die which sets whiptail box to HEIGHT 0. This doesn't show a scrolling window on gpg errors which is problematic with fbwhiptail, not whiptail Signed-off-by: Thierry Laurion <[email protected]>
This was referenced Aug 2, 2024
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This supersedes #1269.... And it works!!!!
Will edit with notes from other PR. After cleaning all those changes that were merged upstream.
TODO:
Write docs ( see NIX_REPRO_NOTES for draft): merged under Heads mainREADME.mdandtarget/qemu.md