Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

busybox build does not reproduce #1636

Closed
JonathonHall-Purism opened this issue Apr 9, 2024 · 1 comment · Fixed by #1661
Closed

busybox build does not reproduce #1636

JonathonHall-Purism opened this issue Apr 9, 2024 · 1 comment · Fixed by #1661
Milestone
reproduciblebuilds

Comments

@JonathonHall-Purism
Copy link
Collaborator

JonathonHall-Purism commented Apr 9, 2024 •
edited
Loading

I compared a CI build of #1630 with my local build, the only remaining reproducibility issue is busybox. All coreboot stages now reproduce, payload differs in the initrd only.

diffoscope identifies busybox. There are a lot of differences in disassembly and relocations but I think they all stem from a difference in string data (which causes more layout differences, etc.):

│ │ │ ├── strings --all --bytes=8 {}
│ │ │ │ @@ -665,15 +665,16 @@
│ │ │ │  []A\A]A^A_
│ │ │ │  h[]A\A]A^A_
│ │ │ │  []A\A]A^A_
│ │ │ │  []A\A]A^A_
│ │ │ │  [A\A]A^A_]
│ │ │ │  AWAVAUATUH
│ │ │ │  []A\A]A^A_
│ │ │ │ -%Y-%m-%d %H:%M:%
│ │ │ │ +   @@@%Y-%m-%d %H:%M:%
│ │ │ │ +NO NAME    
│ │ │ │  JFS1BSFXI
│ │ │ │  hsqsreebb
│ │ │ │   multi-call binary.
│ │ │ │  No help available
│ │ │ │  section header
│ │ │ │  unknown user/group
│ │ │ │  keyword outside section
│ │ │ │ @@ -1969,15 +1970,14 @@
│ │ │ │  can't make FAT32 with >128 sectors/cluster
│ │ │ │  Device '%s':
│ │ │ │  heads:%u, sectors/track:%u, bytes/sector:%u
│ │ │ │  media descriptor:%02x
│ │ │ │  total sectors:%lu, clusters:%u, sectors/cluster:%u
│ │ │ │  FATs:2, sectors/FAT:%u
│ │ │ │  volumeID:%08x, label:'%s'
│ │ │ │ -NO NAME    
│ │ │ │  --More-- 
│ │ │ │  (%u%% of %lu bytes)
│ │ │ │  (Enter:next line Space:next page Q:quit R:show the rest)
│ │ │ │  mount('%s','%s','%s',0x%08lx,'%s'):%d
│ │ │ │  would do mount('%s','%s','%s',0x%08lx,'%s')
│ │ │ │  %s is write-protected, mounting read-only
│ │ │ │  unc=\\%s\%.*s,prefixpath=%s

Some date format string changed slightly, now starts with @@@. NO NAME also moved.

(left is CI, right is my local build, and I did a clean build by deleting build/x86 and install/x86 for this)
@JonathonHall-Purism JonathonHall-Purism added this to the reproduciblebuilds milestone Apr 9, 2024
@tlaurion
Copy link
Collaborator

tlaurion commented Apr 19, 2024 •
edited
Loading

@JonathonHall-Purism some top of the head insight.

modukes/busybox is actually the only one being rebuilt without caring of its .build status from global Makefile rules, bypassing them.

I wonder if this is not the root of the problem, considering that CI rebuilds it multiple times depending of where that board is in the chain of workspace cache transfers.

I think a little rework on modules/busybox (oldest Heads module?) refreshed and respecting proper build recipes could probably resolve the problem altogether with a little bit of investigation. Some timestamps get into that module build.
I think that if both those issues are tackled, that module would be reproducible and we would get rid of:

2024-04-19 19:27:49-04:00 INSTALL-MODULE drivers/usb/storage/usb-storage.ko
/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-strip --preserve-dates --strip-debug -o "/tmp/tmp.PXsyEBOSV2/lib/modules/usb-storage.ko" "/home/user/heads/build/x86/linux-5.10.5/linux-qemu/drivers/usb/storage/usb-storage.ko" 
2024-04-19 19:27:49-04:00 SYMLINK bin/busybox
make -C /home/user/heads/build/x86/busybox-1.36.1 CC="/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/home/user/heads=heads -gno-record-gcc-switches -D__MUSL__ -isystem /home/user/heads/install/x86/include -L/home/user/heads/install/x86/lib " CROSS_COMPILE="/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-" CONFIG_PREFIX="/tmp/tmp.8WKh1F5vTB/bin/.." install | tee -a /home/user/heads/build/x86/log/busybox.log  
make[1]: Entering directory '/home/user/heads/build/x86/busybox-1.36.1'
rm -f .kernelrelease
echo 1.36.1 > .kernelrelease
/home/user/heads/build/x86/busybox-1.36.1/scripts/gen_build_files.sh /home/user/heads/build/x86/busybox-1.36.1 /home/user/heads/build/x86/busybox-1.36.1
make -f scripts/Makefile.build obj=scripts/basic
make -f scripts/Makefile.build obj=applets
make -f scripts/Makefile.build obj=.
make -f scripts/Makefile.build obj=scripts
make -f scripts/Makefile.build obj=applets
make -f scripts/Makefile.build obj=archival
make -f scripts/Makefile.build obj=archival/libarchive
make -f scripts/Makefile.build obj=console-tools
make -f scripts/Makefile.build obj=coreutils
make -f scripts/Makefile.build obj=coreutils/libcoreutils
make -f scripts/Makefile.build obj=debianutils
make -f scripts/Makefile.build obj=klibc-utils
make -f scripts/Makefile.build obj=e2fsprogs
make -f scripts/Makefile.build obj=editors
make -f scripts/Makefile.build obj=findutils
make -f scripts/Makefile.build obj=init
make -f scripts/Makefile.build obj=libbb
make -f scripts/Makefile.build obj=libpwdgrp
make -f scripts/Makefile.build obj=loginutils
make -f scripts/Makefile.build obj=mailutils
make -f scripts/Makefile.build obj=miscutils
make -f scripts/Makefile.build obj=modutils
make -f scripts/Makefile.build obj=networking
make -f scripts/Makefile.build obj=networking/libiproute
make -f scripts/Makefile.build obj=networking/udhcp
make -f scripts/Makefile.build obj=printutils
make -f scripts/Makefile.build obj=procps
make -f scripts/Makefile.build obj=runit
make -f scripts/Makefile.build obj=selinux
make -f scripts/Makefile.build obj=shell
make -f scripts/Makefile.build obj=sysklogd
make -f scripts/Makefile.build obj=util-linux
make -f scripts/Makefile.build obj=util-linux/volume_id
rm -f .old_version
DO_INSTALL_LIBS="" \
	/bin/sh /home/user/heads/build/x86/busybox-1.36.1/applets/install.sh /tmp/tmp.8WKh1F5vTB/bin/.. --symlinks
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/[ -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/[[ -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/arch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ascii -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ash -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/awk -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/base32 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/basename -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bunzip2 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bzip2 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/chattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/chmod -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/clear -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cmp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cpio -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/crc32 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cttyhack -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cut -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/date -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/df -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/diff -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dirname -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dmesg -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/du -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/echo -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/env -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/expr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/factor -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/fallocate -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/false -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/find -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/fold -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/getopt -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/grep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/groups -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/gunzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/gzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/head -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hexdump -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hexedit -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hostid -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/id -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/install -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/kill -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/killall -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/less -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/link -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ln -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ls -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsof -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsscsi -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsusb -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lzma -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/md5sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mkdir -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mkfifo -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mknod -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mktemp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/more -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mount -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mv -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nl -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nproc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nslookup -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/paste -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/patch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pgrep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pidof -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ping -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pkill -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/printf -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ps -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pwd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/readlink -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/realpath -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/reset -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/resume -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/rm -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/rmdir -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sed -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/seq -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setfattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setpriv -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setserial -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setsid -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sh -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha1sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha256sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha3sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha512sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/shred -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sleep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sort -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ssl_client -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/stat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/strings -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/stty -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sync -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tail -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tar -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tee -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/test -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tftp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/time -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/top -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/touch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tree -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/true -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/truncate -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tsort -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tty -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/umount -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/uname -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/uniq -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unlzma -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unxz -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/usleep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/vi -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/wc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/wget -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/which -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xargs -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xxd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xz -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/zcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/arp -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/blkid -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/blockdev -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/chroot -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/devmem -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fdisk -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fsck -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fsfreeze -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/hwclock -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cdetect -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cdump -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cget -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cset -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ifconfig -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/insmod -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ip -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/killall5 -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/loadkmap -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/losetup -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/lsmod -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/mkdosfs -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/mkfs.vfat -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/modinfo -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ntpd -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/partprobe -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/route -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/seedrng -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/sysctl -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/udhcpc -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/vconfig -> ../bin/busybox


--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------

make[1]: Leaving directory '/home/user/heads/build/x86/busybox-1.36.1'

Which spams console on every local rebuild when only bash/ash scripts changed. This should vanish, and at the same time, this issue.

@tlaurion tlaurion linked a pull request May 9, 2024 that will close this issue
Move to nix buildstack (and nix develop produced docker image used under CircleCI) #1661
Merged
3 tasks
@tlaurion tlaurion mentioned this issue May 9, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
reproduciblebuilds
Development

Successfully merging a pull request may close this issue.

Move to nix buildstack (and nix develop produced docker image used under CircleCI) tlaurion/heads
2 participants
@tlaurion @JonathonHall-Purism