Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bcache partition not registered in initramfs (with raid probes from udev) #3

Open
mattibal opened this issue Oct 20, 2013 · 12 comments

Comments

@mattibal
Copy link

Hi,

I installed your bcache-tools package from your ppa to use bcache for the root partition of my Ubuntu 13.10 install.
I have a bcache backing partition on my HDD at /dev/sda1, and a bcache caching partition on my SSD at /dev/sdb1.
On the top of /dev/bcache0, I have a btrfs filesystem, which is the root filesystem of my Ubuntu install.

The problem is: when I try to start Ubuntu, the boot procedure stops showing me the initramfs busybox shell, because it can't find the root btrfs filesystem which is inside bcache.
By looking at the busybox shell, I have seen that the bcache partition on my SSD is registered, but the one on my HDD not, so the /dev/bcache0 device is not created.
So I tried to launch in the initramfs shell: "echo /dev/sda1 > /sys/fs/bcache/register" and "exit", and then the system continued to boot perfectly.

As a workaround I added the line "echo /dev/sda1 > /sys/fs/bcache/register" at the end of /lib/udev/bcache-register. Now it works for me, but of course this is not a real solution :)

@g2p
Copy link
Owner

g2p commented Oct 20, 2013

Is it the latest version from the PPA? Can you run blkid on that device, if it finds something non-bcache you'll have to remove it with wipefs (-a to confirm). Recent make-bcache should prevent you from getting into this situation, if it doesn't it's a bug.

@mattibal
Copy link
Author

Yes, it's the latest version from PPA, I installed them 2 days ago.

This is the output of blkid from the distro running on btrfs on the top of bcache:
mattia@mattia-laptop:$ sudo blkid /dev/bcache0
/dev/bcache0: UUID="b2ca33df-5283-4a2f-a6df-76de81c277fb" UUID_SUB="95fd45b1-5de5-4ccb-8f48-15cf75944764" TYPE="btrfs"
mattia@mattia-laptop:
$ sudo blkid /dev/sda1
mattia@mattia-laptop:$ sudo blkid /dev/sdb1
mattia@mattia-laptop:
$
(it doesn't give any output for the last two, maybe because are in use by bcache, if you want I will try from the Ubuntu live)

To make the bcache partitions, I did this:

  • created with gparted an ext4 partition on the HDD (/dev/sda1)
  • created with gparted an ext4 partition on the SSD (/dev/sdb1)
  • umounted both partition
  • wipefs -a /dev/sda1
  • wipefs -a /dev/sdb1
  • make-bcache -B /dev/sda1 -C /dev/sdb1

Let me know if you need more info.
Thanks!

@g2p
Copy link
Owner

g2p commented Oct 21, 2013

Le 21/10/2013 01:53, Mattia Baldani a écrit :

Yes, it's the latest version from PPA, I installed them 2 days ago.

This is the output of blkid from the distro running on btrfs on the top
of bcache:
mattia@mattia-laptop:$ sudo blkid /dev/bcache0
/dev/bcache0: UUID="b2ca33df-5283-4a2f-a6df-76de81c277fb"
UUID_SUB="95fd45b1-5de5-4ccb-8f48-15cf75944764" TYPE="btrfs"
mattia@mattia-laptop:
$ sudo blkid /dev/sda1
mattia@mattia-laptop:~$ sudo blkid /dev/sdb1
(it doesn't give any output for the last two, maybe because are in use
by bcache, if you want I will try from the Ubuntu live)

(bcache isn't recognised without a beta release of blkid)

To make the bcache partitions, I did this:

  • created with gparted an ext4 partition on the HDD (/dev/sda1)
  • created with gparted an ext4 partition on the SSD (/dev/sdb1)
  • umounted both partition
  • wipefs -a /dev/sda1
  • wipefs -a /dev/sdb1
  • make-bcache -B /dev/sda1 -C /dev/sdb1

Let me know if you need more info.
Thanks!

Seems like the udev rules are broken then. They have changed a bit
recently. It would be helpful if you could bisect from 1.0.1-1, old
versions are available here:

https://launchpad.net/~g2p/+archive/storage/+packages?field.name_filter=bcache&field.status_filter=&field.series_filter=saucy

@mattibal
Copy link
Author

I tried with version 1.0.1-1 and still doesn't work...

@g2p
Copy link
Owner

g2p commented Oct 21, 2013

I'm out of ideas right now. Either something is preventing the rule from running on the backing device, or something is keeping the device busy just when the kernel is trying to open it. Maybe the latter case would be mentioned in dmesg. Also, installing the package only rebuilds the newest initramfs, in the unlikely case you're booting into an older kernel it would still be running the 1.0.4-1 rules.

@mattibal
Copy link
Author

I tried to replace register_quiet with register in /lib/udev/bcache-register, in order to see more dmesg messages, and this is the result:

Oct 21 02:35:40 mattia-laptop kernel: [ 2.499854] [drm] Initialized nouveau 1.1.1 20120801 for 0000:01:00.0 on minor 0
Oct 21 02:35:40 mattia-laptop kernel: [ 2.624100] ata6: SATA link down (SStatus 0 SControl 300)
Oct 21 02:35:40 mattia-laptop kernel: [ 2.681160] bio: create slab at 1
Oct 21 02:35:40 mattia-laptop kernel: [ 2.774847] bcache: bch_journal_replay() journal replay done, 336 keys in 54 entries, seq 32243
Oct 21 02:35:40 mattia-laptop kernel: [ 2.775033] bcache: register_cache() registered cache device sdb1
Oct 21 02:35:40 mattia-laptop kernel: [ 52.059216] bcache: register_bdev() registered backing device sda1
Oct 21 02:35:40 mattia-laptop kernel: [ 52.062196] bcache: bch_cached_dev_attach() Caching sda1 as bcache0 on set 90630c89-402c-43bd-b5f3-41e67181d8b8
Oct 21 02:35:40 mattia-laptop kernel: [ 52.968041] raid6: sse2x1 3891 MB/s

Unfortunately nothing of interesting... the cache device sdb1 is registered automatically, the backing device sda1 has been registered manually by me from the initramfs shell.

I've used the same kernel for 1.0.1-1 version.

I don't know exactly how the udev thing works, but I think the /lib/udev/bcache-register script is launched only for the cache sdb1 device, and not for the backing sda1. By adding a line in that script that registers sda1, everything works fine...

@g2p
Copy link
Owner

g2p commented Oct 21, 2013

This page has some info on debugging udev:
https://wiki.ubuntu.com/DebuggingUdev#Useful_things

You already have a /var/log/udev, but I doubt the sda1 entry has the smoking gun.
You could sudoedit /usr/share/initramfs-tools/scripts/init-top/udev, add --debug and redirections to /dev/.udev.debug to the udevd line, and rebuild the initramfs (update-initramfs -u -k all) (slightly modified from the wiki instructions because the .initramfs directory might not exist).

@mattibal
Copy link
Author

In /var/log/udev I'm not able to see anything interesting.
I tried the udev --debug thing but I got an empty log file... udev hates me :)

@g2p
Copy link
Owner

g2p commented Oct 21, 2013

It probably landed in /var/log/syslog, I've updated the wiki page.

@mattibal
Copy link
Author

I finally got the udev log from dmesg, this is the point where it register /dev/sdb1 but not the backing /dev/sda1 (look for the line in bold):

[ 2.607431] systemd-udevd[111]: LINK 'disk/by-label/Ubuntu\x2013.10\x20amd64' /lib/udev/rules.d/60-persistent-storage.rules:79
[ 2.607446] systemd-udevd[111]: LINK 'disk/by-id/wwn-0x50014ee25a1a21e7' /lib/udev/rules.d/60-persistent-storage.rules:82
[ 2.607467] systemd-udevd[111]: handling device node '/dev/sda', devnum=b8:0, mode=0600, uid=0, gid=0
[ 2.607481] systemd-udevd[111]: preserve permissions /dev/sda, 060600, uid=0, gid=0
[ 2.607500] systemd-udevd[111]: creating symlink '/dev/block/8:0' to '../sda'
[ 2.607530] systemd-udevd[111]: creating link '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619' to '/dev/sda'
[ 2.607538] systemd-udevd[111]: creating symlink '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619' to '../../sda'
[ 2.607569] systemd-udevd[111]: creating link '/dev/disk/by-id/wwn-0x50014ee25a1a21e7' to '/dev/sda'
[ 2.607575] systemd-udevd[111]: creating symlink '/dev/disk/by-id/wwn-0x50014ee25a1a21e7' to '../../sda'
[ 2.607610] systemd-udevd[111]: creating link '/dev/disk/by-label/Ubuntu\x2013.10\x20amd64' to '/dev/sda'
[ 2.607617] systemd-udevd[111]: creating symlink '/dev/disk/by-label/Ubuntu\x2013.10\x20amd64' to '../../sda'
[ 2.607715] systemd-udevd[111]: created db file '/run/udev/data/b8:0' for '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'
[ 2.607722] systemd-udevd[111]: adding watch on '/dev/sda'
[ 2.607785] systemd-udevd[111]: created db file '/run/udev/data/b8:0' for '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'
[ 2.607814] systemd-udevd[111]: passed -1 bytes to netlink monitor 0xf3ae60
[ 2.607829] systemd-udevd[111]: seq 1492 processed with 0
[ 2.607867] systemd-udevd[105]: seq 1492 done with 0
[ 2.607908] systemd-udevd[105]: passed 254 bytes to netlink monitor 0xf1d3a0
[ 2.607923] systemd-udevd[105]: passed 254 bytes to netlink monitor 0xf1d3a0
[ 2.607958] systemd-udevd[108]: seq 1493 running
[ 2.607973] systemd-udevd[109]: seq 1494 running
[ 2.608072] systemd-udevd[108]: no db file to read /run/udev/data/b8:1: No such file or directory
[ 2.608075] systemd-udevd[109]: no db file to read /run/udev/data/b8:2: No such file or directory
[ 2.608120] systemd-udevd[108]: device 0xf22b30 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'
[ 2.608125] systemd-udevd[109]: device 0xf22a50 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'
[ 2.608203] systemd-udevd[108]: device 0xf22b30 filled with db file data
[ 2.608209] systemd-udevd[109]: device 0xf22a50 filled with db file data
[ 2.608231] systemd-udevd[108]: LINK 'disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part1' /lib/udev/rules.d/60-persistent-storage.rules:44
[ 2.608240] systemd-udevd[109]: LINK 'disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part2' /lib/udev/rules.d/60-persistent-storage.rules:44
[ 2.608271] systemd-udevd[108]: device 0xf252d0 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0'
[ 2.608280] systemd-udevd[109]: device 0xf25020 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0'
[ 2.608290] systemd-udevd[108]: device 0xf2f1b0 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0'
[ 2.608299] systemd-udevd[109]: device 0xf257c0 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0'
[ 2.608307] systemd-udevd[108]: device 0xf2f990 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0'
[ 2.608314] systemd-udevd[109]: device 0xf2f570 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0'
[ 2.608322] systemd-udevd[108]: device 0xf30160 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1'
[ 2.608329] systemd-udevd[109]: device 0xf2fd40 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1'
[ 2.608337] systemd-udevd[108]: device 0xf30920 has devpath '/devices/pci0000:00/0000:00:1f.2'
[ 2.608343] systemd-udevd[109]: device 0xf30500 has devpath '/devices/pci0000:00/0000:00:1f.2'
[ 2.608352] systemd-udevd[108]: device 0xf31160 has devpath '/devices/pci0000:00'
[ 2.608356] systemd-udevd[109]: device 0xf30cb0 has devpath '/devices/pci0000:00'
[ 2.608365] systemd-udevd[108]: IMPORT builtin 'blkid' /lib/udev/rules.d/60-persistent-storage.rules:72
[ 2.608368] systemd-udevd[109]: IMPORT builtin 'blkid' /lib/udev/rules.d/60-persistent-storage.rules:72
[ 2.608397] systemd-udevd[108]: probe /dev/sda1 raid offset=0
[ 2.608449] systemd-udevd[109]: probe /dev/sda2 raid offset=0
[ 2.663665] systemd-udevd[107]: 'bcache-register /dev/sdb1' [164] exit with return code 0
[ 2.663683] systemd-udevd[107]: adding watch on '/dev/sdb1'
[ 2.663810] systemd-udevd[107]: created db file '/run/udev/data/b8:17' for '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb1'
[ 2.663829] systemd-udevd[107]: passed -1 bytes to netlink monitor 0xf2cdc0
[ 2.663844] systemd-udevd[107]: seq 1506 processed with 0
[ 2.663886] systemd-udevd[105]: seq 1506 done with 0
[ 2.701796] systemd-udevd[108]: LINK 'disk/by-label/Ubuntu\x2013.10\x20amd64' /lib/udev/rules.d/60-persistent-storage.rules:79
[ 2.701807] systemd-udevd[108]: LINK 'disk/by-id/wwn-0x50014ee25a1a21e7-part1' /lib/udev/rules.d/60-persistent-storage.rules:83
[ 2.701820] systemd-udevd[108]: handling device node '/dev/sda1', devnum=b8:1, mode=0600, uid=0, gid=0
[ 2.701826] systemd-udevd[108]: preserve permissions /dev/sda1, 060600, uid=0, gid=0
[ 2.701838] systemd-udevd[108]: creating symlink '/dev/block/8:1' to '../sda1'
[ 2.701854] systemd-udevd[108]: creating link '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part1' to '/dev/sda1'
[ 2.701862] systemd-udevd[108]: creating symlink '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part1' to '../../sda1'
[ 2.701889] systemd-udevd[108]: creating link '/dev/disk/by-id/wwn-0x50014ee25a1a21e7-part1' to '/dev/sda1'
[ 2.701896] systemd-udevd[108]: creating symlink '/dev/disk/by-id/wwn-0x50014ee25a1a21e7-part1' to '../../sda1'
[ 2.701925] systemd-udevd[108]: found 'b8:0' claiming '/run/udev/links/\x2fdisk\x2fby-label\x2fUbuntu\x5cx2013.10\x5cx20amd64'
[ 2.701949] systemd-udevd[108]: device 0xf42c80 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'
[ 2.702029] systemd-udevd[108]: device 0xf42c80 filled with db file data
[ 2.702042] systemd-udevd[108]: creating link '/dev/disk/by-label/Ubuntu\x2013.10\x20amd64' to '/dev/sda1'
[ 2.702050] systemd-udevd[108]: atomically replace '/dev/disk/by-label/Ubuntu\x2013.10\x20amd64'
[ 2.702122] systemd-udevd[108]: created db file '/run/udev/data/b8:1' for '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1'
[ 2.702128] systemd-udevd[108]: adding watch on '/dev/sda1'
[ 2.702182] systemd-udevd[108]: created db file '/run/udev/data/b8:1' for '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1'
[ 2.702200] systemd-udevd[108]: passed -1 bytes to netlink monitor 0xf238e0
[ 2.702209] systemd-udevd[108]: seq 1493 processed with 0
[ 2.702243] systemd-udevd[105]: seq 1493 done with 0
[ 2.726625] systemd-udevd[109]: LINK 'disk/by-uuid/7618215d-ac30-4834-99b4-ae331fd1423f' /lib/udev/rules.d/60-persistent-storage.rules:78
[ 2.726634] systemd-udevd[109]: LINK 'disk/by-label/Ubuntu\x2013.10\x20amd64' /lib/udev/rules.d/60-persistent-storage.rules:79
[ 2.726642] systemd-udevd[109]: LINK 'disk/by-id/wwn-0x50014ee25a1a21e7-part2' /lib/udev/rules.d/60-persistent-storage.rules:83
[ 2.726654] systemd-udevd[109]: handling device node '/dev/sda2', devnum=b8:2, mode=0600, uid=0, gid=0
[ 2.726660] systemd-udevd[109]: preserve permissions /dev/sda2, 060600, uid=0, gid=0
[ 2.726669] systemd-udevd[109]: creating symlink '/dev/block/8:2' to '../sda2'
[ 2.726685] systemd-udevd[109]: creating link '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part2' to '/dev/sda2'
[ 2.726692] systemd-udevd[109]: creating symlink '/dev/disk/by-id/ata-WDC_WD5000BEKT-00KA9T0_WD-WXK1A60F4619-part2' to '../../sda2'
[ 2.726716] systemd-udevd[109]: creating link '/dev/disk/by-id/wwn-0x50014ee25a1a21e7-part2' to '/dev/sda2'
[ 2.726725] systemd-udevd[109]: creating symlink '/dev/disk/by-id/wwn-0x50014ee25a1a21e7-part2' to '../../sda2'
[ 2.726758] systemd-udevd[109]: found 'b8:1' claiming '/run/udev/links/\x2fdisk\x2fby-label\x2fUbuntu\x5cx2013.10\x5cx20amd64'
[ 2.726780] systemd-udevd[109]: device 0xf430a0 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1'
[ 2.726861] systemd-udevd[109]: device 0xf430a0 filled with db file data
[ 2.726870] systemd-udevd[109]: found 'b8:0' claiming '/run/udev/links/\x2fdisk\x2fby-label\x2fUbuntu\x5cx2013.10\x5cx20amd64'
[ 2.726887] systemd-udevd[109]: device 0xf430a0 has devpath '/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda'

@rolffokkens
Copy link
Contributor

The dmesg lines suggest that /dev/sda1 and /dev/sda2 are considered raid,
which doesn't seem right.

Van: Mattia Baldani [email protected]
Datum: Monday, October 21, 2013 12:31 PM
Aan: g2p/bcache-tools [email protected]
Onderwerp: Re: [bcache-tools] bcache partition not registered in
initramfs (#3)

I finally got the udev log from dmesg, this is the point where it register
/dev/sdb1 but not the backing /dev/sda1:
[ 2.608397] systemd-udevd[108]: probe /dev/sda1 raid offset=0
[ 2.608449] systemd-udevd[109]: probe /dev/sda2 raid offset=0
[ 2.663665] systemd-udevd[107]: 'bcache-register /dev/sdb1' [164] exit
with return code 0

@mattibal
Copy link
Author

Ok thanks, I imagine it's an udev bug... I never had set up a raid in these disks I'm using

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants