Occasional build failure #50
Labels
Comments
|
Some other logs from @Thread974 collected on different host: |
|
Unfortunately have no enough information to close or debug :-( |
|
I haven't tried for a while, but I can give it another try and let you know if it still happens. |
|
It occured several times this week while building armhf images. I notice it while doing ospacks and images, rather that doing only images. |
|
So with some recent improvement to fakemachine to speed up the 9pfs stuff this actually tends to happen more often. Which made it reproducible enough for some more analysis/testing. The problem is that we currently assume in debos that after doing the partitioning we can use udev settle to wait for the device node to be create.. Unfortunately while you would expect adding a partitoin to just cause a single uevent adding the new device, it seems reality isn't that simple. What actually comes from the kernel is:
So it seems that when the kernel is asked to recheck the partition table it seemingly drops all current partition device and re-adds/creates them. In some artifical testing in a fakemachine udevmonitor also shows there is a gap of about 16 miliseconds between the first add event on the new partition and the second round of remove/add events. So the reason for this race is that there is a small window where the devices files are removed and recreated by udev, if that just happens when a filesystem is created or tried to be mounted you get the issues above. |
|
After a bit of peering at the kernel code and not understand why we're getting multiple sequences of events i did a bit more debugging.. It turns out udev is trying to be too helpful and will trigger a rescan of the partition table iff the block device itsef triggers an inotify IN_CLOSE_WRITE and it can get an exclusive lock on the device (e.g. no partitions mounted). When using parted this isn't useful, as parted already tell the kernel about the new partition (triggering an add uevent) and just adds confusion as seen above.. Testing with flock shows that indeed prevents it from happening and only and only a single add uevent for the new partition occurs, which is what we'd want to clear this up properly. |
sjoerdsimons
added a commit
that referenced
this issue
Jun 15, 2018
Every so often we've seen races where seemingly the partition device disappeared when trying to use it. These became a bit more frequent after some speedups in fakemachine. Digging deeper it turns out that udev helpfully ask the kernel to rescan the partition table when it gets told via inotify that something might have written to it. As we're using parted, which already tells the kernel about partitions it created, this gets rather confusion. The new partition device would pop up (triggered by parted), parted would close the disk device node; This then would notify udev, which asked the kernel to rescan the partition table on the device *again*, which in turn causes all the partition device nodes to disappear and re-appear, leaving a nice window for things to go horribly horribly wrong. Ofcourse this all happens asynchronously, so there is really no way to know everything settled down.. Luckily there is one upside to this depressing story, udev only does this when it can get an exclusive lock on the disk device (e.g. none of the partitions are mounted and nobody else holds an exclusive lock). So we can avoid all this madness by simply holding an exclusive device lock and only releasing it after mounting has happened. As one last unexpected side-effect, while we do hold the exclusive lock it turns out udev also does not setup the various symlinks. So instead for the partition device use the canonical device names rather then the aliases. On the bright side those device nodes are generated directly by the kernel with devtmpfs, so there is no need to wait for udev to have done its processing anymore. Closes #50 Signed-off-by: Sjoerd Simons <[email protected]>
sjoerdsimons
added a commit
that referenced
this issue
Jun 15, 2018
Every so often we've seen races where seemingly the partition device disappeared when trying to use it. These became a bit more frequent after some speedups in fakemachine. Digging deeper it turns out that udev helpfully ask the kernel to rescan the partition table when it gets told via inotify that something might have written to it. As we're using parted, which already tells the kernel about partitions it created, this gets rather confusion. The new partition device would pop up (triggered by parted), parted would close the disk device node; This then would notify udev, which asked the kernel to rescan the partition table on the device *again*, which in turn causes all the partition device nodes to disappear and re-appear, leaving a nice window for things to go horribly horribly wrong. Ofcourse this all happens asynchronously, so there is really no way to know everything settled down.. Luckily there is one upside to this depressing story, udev only does this when it can get an exclusive lock on the disk device (e.g. none of the partitions are mounted and nobody else holds an exclusive lock). So we can avoid all this madness by simply holding an exclusive device lock and only releasing it after mounting has happened. As one last unexpected side-effect, while we do hold the exclusive lock it turns out udev also does not setup the various symlinks. So instead for the partition device use the canonical device names rather then the aliases. On the bright side those device nodes are generated directly by the kernel with devtmpfs, so there is no need to wait for udev to have done its processing anymore. Closes #50 Signed-off-by: Sjoerd Simons <[email protected]>
sjoerdsimons
added a commit
that referenced
this issue
Jun 15, 2018
Every so often we've seen races where seemingly the partition device disappeared when trying to use it. These became a bit more frequent after some speedups in fakemachine. Digging deeper it turns out that udev helpfully ask the kernel to rescan the partition table when it gets told via inotify that something might have written to it. As we're using parted, which already tells the kernel about partitions it created, this gets rather confusion. The new partition device would pop up (triggered by parted), parted would close the disk device node; This then would notify udev, which asked the kernel to rescan the partition table on the device *again*, which in turn causes all the partition device nodes to disappear and re-appear, leaving a nice window for things to go horribly horribly wrong. Ofcourse this all happens asynchronously, so there is really no way to know everything settled down.. Luckily there is one upside to this depressing story, udev only does this when it can get an exclusive lock on the disk device (e.g. none of the partitions are mounted and nobody else holds an exclusive lock). So we can avoid all this madness by simply holding an exclusive device lock and only releasing it after mounting has happened. As one last unexpected side-effect, while we do hold the exclusive lock it turns out udev also does not setup the various symlinks. So instead for the partition device use the canonical device names rather then the aliases. On the bright side those device nodes are generated directly by the kernel with devtmpfs, so there is no need to wait for udev to have done its processing anymore. Closes #50 Signed-off-by: Sjoerd Simons <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm using go-debos and I am getting build failure.
It does not always occur. I'm on ubuntu and have a couple patches.
The text was updated successfully, but these errors were encountered: