ZFS userdata has disappeared without any errors after the reboot #16614

barl0g · 2024-10-07T09:41:27Z

barl0g
Oct 7, 2024

After the reboot, part of the data on tank/rpool has disappeared. There are no tank/rpool/USERDATA/root_xxxx11 and tank/rpool/USERDATA/user_xxxx11. There are no error messages in the zpool status or scrub. The smartctl status has "passed" for all disks.

Before the reboot, I experimented with incremental backups using snapshot send.

I have backups, but not all of the data is included in the backups.

It is Ubuntu 22.04.4 LTS.

# zpool list
no pools available

# zpool import -lR /mnt tank
Enter passphrase for 'tank/rpool':
1 / 1 keys successfully loaded

# zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 00:19:03 with 0 errors on Mon Oct  7 07:33:16 2024
remove: Removal of vdev 1 copied 7.19G in 0h1m, completed on Wed Apr 10 18:05:33 2024
	122K memory used for removed device mappings
config:

	NAME                                    STATE     READ WRITE CKSUM
	tank                                    ONLINE       0     0     0
	  66969040-8f95-6f4d-aca3-dfe34ff2d1f7  ONLINE       0     0     0
	  sdb1                                  ONLINE       0     0     0
	cache
	  sdc3                                  ONLINE       0     0     0

errors: No known data errors

# ls /mnt/home/user/
mnt  tmp

Where is my files?!

# zfs mount -a

# ls /mnt/home/user/
mnt  tmp

# zfs list | grep USERDATA
tank/rpool/USERDATA                                                                               168K   374G      168K  /mnt

# zfs list -t snapshot | grep USERDATA
tank/rpool/[email protected]_20240908                                                                                     0B      -      168K  -
tank/rpool/[email protected]_20241006                                                                                 0B      -      168K  -

# zpool history tank|tail
2024-10-06.07:05:50 zpool import -N tank
2024-10-06.07:06:12 zfs load-key tank/rpool
2024-10-06.06:16:00 zfs snapshot -r tank/[email protected]_20241006
2024-10-06.06:59:35 zfs send --raw -Rvi tank/[email protected]_20240908 tank/[email protected]_20241006
2024-10-07.08:34:09 zpool export tank
2024-10-07.08:40:47 zpool import -lR /mnt tank

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   492G   106G   386G        -         -    19%    21%  1.00x    ONLINE  /mnt

# df -h
--cut--
tank/rpool/ROOT/ubuntu_xxxx                          428G   58G  371G  14% /mnt
--cut--

About 50GB of data in tank/rpool/USERDATA/user_xxxx11 has disappeared.

In my backup I save zfs list

$ cat /sdb2/info/zfs_list | grep USERDATA
tank/rpool/USERDATA                                                                              57.6G   352G      168K  /
tank/rpool/USERDATA/root_a6ehn3                                                                  36.6M   352G     36.6M  /root
tank/rpool/USERDATA/user_a6ehn3                                                                  57.5G   352G     57.5G  /home/user

In the current system there is no user's data

$ sudo zfs list | grep USERDATA
tank/rpool/USERDATA                                                                               168K   370G      168K  /mnt

Any help would be appreciated.

amotin · 2024-10-07T14:54:58Z

amotin
Oct 7, 2024
Collaborator

Make sure you are really looking at the proper place: that you have written the data where you think you did, not under the mount point of previously not mounted dataset, or that you have really mounted the dataset where do you think you have.

1 reply

barl0g Oct 8, 2024
Author

I understand the overmounting case, and it's not that issue.

IvanVolosyuk · 2024-10-07T23:00:48Z

IvanVolosyuk
Oct 7, 2024

Pretty much what Alexander said, something can be mounted on top of your data. Try locating the data using:

zpool ist
zfs list

The common mistake is to have the data under a mounted filesystem, may be:

zpool umount tank/rpool/USERDATA

and see if it's there.

1 reply

barl0g Oct 8, 2024
Author

zfs list or zfs list -t snapshots doesn't show my user data :(

ofthesun9 · 2024-10-08T06:14:39Z

ofthesun9
Oct 8, 2024

Do you have the zsys package installed and running ? This Ubuntu package considers USERDATA location as reserved and is handling datasets stored there with its own rules (including deletion)

ubuntu/zsys#218

1 reply

barl0g Oct 8, 2024
Author

Yes, the zsys package is installed and running, but as far as I know, zsys typically only creates and deletes snapshots.

barl0g · 2024-10-08T08:51:38Z

barl0g
Oct 8, 2024
Author

In my backup I save zfs list

$ cat /sdb2/info/zfs_list | grep USERDATA
tank/rpool/USERDATA                                                                              57.6G   352G      168K  /
tank/rpool/USERDATA/root_a6ehn3                                                                  36.6M   352G     36.6M  /root
tank/rpool/USERDATA/user_a6ehn3                                                                  57.5G   352G     57.5G  /home/user

In the current system there is no user's data

$ sudo zfs list | grep USERDATA
tank/rpool/USERDATA                                                                               168K   370G      168K  /mnt

0 replies

luckylinux · 2024-10-09T04:40:32Z

luckylinux
Oct 9, 2024

@barl0g: ZFS Version ?

Although in my case I always build ZFS from Source (openzfs) since Ubuntu at some point did some pretty nasty Things using ZFS (2 Years ago ??) resulting in some CRAZY Bug (which thankfully did NOT hit me).

Another thing to consider is: unmount your mountpoint (umount /my/dir or zfs umount rpool/my/dir) do a chattr +i /my/dir), then remount it (mount /my/dir or zfs mount rpool/my/dir).

It's possible that the mount Order determined by ZFS Systemd Generator was wrong at some point or, for whatever reason, one file got created in the wrong Place then ZFS refused to mount the Filesystem etc.

At least in my case in old releases of ZFS where chattr +i /my/dir was NOT supported or would lead to Errors), child dataset would sometimes mount and sometimes not, resulting in "weird" Things, about where Data would be stored.

Furthermore: your tank/rpool/USERDATA has / as a mountpoint (!!!). That definitively seems weird ... In my case rpool/USERDATA is set to mountpoint=none since root (/root) and normal user <user> (/home/<user>) are under different "root" mounts. You could default to mountpoint=/home but mountpoint=/` definitively looks weird to me !

More importantly: your tank Pool is a Striped VDEV (RAID-0), NOT a Mirror VDEV (RAID-1) !!! Is this intentional ?

About why at some point the Children Datasets disappear (tank/rpool/USERDATA/root_a6ehn3 and tank/rpool/USERDATA/user_a6ehn3) I'm not too sure about ... It definitively looks weird 😮.

Are you sure you didn't move these yourself to somewhere else ?

Maybe try zpool events to see a bit more into it ....

Do a full zpool history as well and look a bit more into the Past, maybe something shows up besides the 6 or so Lines you provided ?

EDIT 1: if you want proper Help I suggest you give the full output of zfs list so that we can see if your "User Data" went somewhere else (for whatever reason). Right now we have to guess and interpolate (you seem to still have 450 GB or so on the Pool, so I guess there is some User Data within those, right ?)

Like ... you provide sometimes tank/rpool/USERDATA Info and sometimes tank/rpool/ROOT/ubuntu_xxxx which is the ROOT Partition (/).

It's really difficult to follow to be honest.

A zfs list would be far more clear. Of course obscure / anonymize the Output if you have any sensitive Information there, but please show all the Dataset infos to see what really is going on.

1 reply

barl0g Oct 9, 2024
Author

@barl0g: ZFS Version ?

zfs --version
zfs-2.1.5-1ubuntu6~22.04.4
zfs-kmod-2.2.2-0ubuntu9

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

Although in my case I always build ZFS from Source (openzfs) since Ubuntu at some point did some pretty nasty Things using ZFS (2 Years ago ??) resulting in some CRAZY Bug (which thankfully did NOT hit me).

I'll take note of this

Another thing to consider is: unmount your mountpoint (umount /my/dir or zfs umount rpool/my/dir) do a chattr +i /my/dir), then remount it (mount /my/dir or zfs mount rpool/my/dir).

I can't do this now. I restored data from a backup. But it doesn't seem like it would help. Because zfs list would show user data

It's possible that the mount Order determined by ZFS Systemd Generator was wrong at some point or, for whatever reason, one file got created in the wrong Place then ZFS refused to mount the Filesystem etc.

Ouch!

At least in my case in old releases of ZFS where chattr +i /my/dir was NOT supported or would lead to Errors), child dataset would sometimes mount and sometimes not, resulting in "weird" Things, about where Data would be stored.

Furthermore: your tank/rpool/USERDATA has / as a mountpoint (!!!). That definitively seems weird ... In my case rpool/USERDATA is set to mountpoint=none since root (/root) and normal user <user> (/home/<user>) are under different "root" mounts. You could default to mountpoint=/home but mountpoint=/` definitively looks weird to me !

This is a very true point. Really weird and could be the reason I lost my data. However, this is what the Ubuntu installer did for me on both systems. I fixed it on local system.

sudo zfs set mountpoint=none tank/rpool/USERDATA

More importantly: your tank Pool is a Striped VDEV (RAID-0), NOT a Mirror VDEV (RAID-1) !!! Is this intentional ?
Yes, that interntional.

About why at some point the Children Datasets disappear (tank/rpool/USERDATA/root_a6ehn3 and tank/rpool/USERDATA/user_a6ehn3) I'm not too sure about ... It definitively looks weird 😮.

Yes :(

Are you sure you didn't move these yourself to somewhere else ?

Yes.

Maybe try zpool events to see a bit more into it ....
I watched them.

Do a full zpool history as well and look a bit more into the Past, maybe something shows up besides the 6 or so Lines you provided ?
I provided the log before the last successful start of the system.
But there is nothing in the full log that would help.

EDIT 1: if you want proper Help I suggest you give the full output of zfs list so that we can see if your "User Data" went somewhere else (for whatever reason). Right now we have to guess and interpolate (you seem to still have 450 GB or so on the Pool, so I guess there is some User Data within those, right ?)

I can't do this now.

Like ... you provide sometimes tank/rpool/USERDATA Info and sometimes tank/rpool/ROOT/ubuntu_xxxx which is the ROOT Partition (/).

It's really difficult to follow to be honest.

A zfs list would be far more clear. Of course obscure / anonymize the Output if you have any sensitive Information there, but please show all the Dataset infos to see what really is going on.

barl0g · 2024-10-09T05:21:06Z

barl0g
Oct 9, 2024
Author

I gave up and restored my system from a backup. I'm planing to disable zsys now.

0 replies

barl0g · 2024-10-09T11:36:33Z

barl0g
Oct 9, 2024
Author

After restoring from a backup, I lost tank/rpool/USERDATA/root_a6ehn3, but tank/rpool/USERDATA/user_a6ehn3 still exists. It's not a huge issue because I have other backups, but it's very interesting. Should I create tank/rpool/USERDATA/root_a6ehn3 for /root?

EDIT 1: To clarify, the first backup I mentioned is a ZFS image, while the second is a simple archive.
EDIT 2: My backup script creates a file with the output of 'zfs list', so I know that tank/rpool/USERDATA/root_a6ehn3 must exist.

5 replies

luckylinux Oct 10, 2024

Well it's definitively weird ... But difficult to tell without having a look at the entire script.

Like ... Are you sure you don't maybe just save the output of zfs list then loop over /home and find the matching Dataset there, and only backup that ?

When I do backup I typically do zfs snapshot -r rpool@YYYYMMDD-Some-Name. Or if you want ALL User Data you could do zfs snapshot -r rpool/USERDATA@YYYYMMDD-Some-Name instead.

But once again, you are asking us to debug blindly. We have NO IDEA if you did wrong in the Backup Process, or something is acting up during the Restore Process. We'd have to see Both Scripts to comment better.

Also no clue what is a "ZFS Image". Do you mean a "ZFS Snapshot" ? Which you backed up as zfs send -Rv rpool/USERDATA@YYYYMMDD-Some-Name | zfs receive -du somewhere/else/USERDATA@YYYYMMDD-Some-Name on another ZFS Pool (built from either a Physical VDEV or some .img File-based VDEV) ?

"A Simple Archive" is also not very clear. How did you build that ? If you just did a tar or similar of /home, of course /root will NOT be included in that. Similarly, if for whatever Reason (like the Mounting order of ZFS Generators for instance) some Filesystem did NOT mount correctly, then your tar would also NOT contain the Backup of that relevant Mountpoint ...

barl0g Oct 10, 2024
Author

Well it's definitively weird ... But difficult to tell without having a look at the entire script.

Like ... Are you sure you don't maybe just save the output of zfs list then loop over /home and find the matching Dataset there, and only backup that ?

I'm backing up the entire pool

rpool_initial_search_pattern="tank/[email protected]_"
rpool_snap="${rpool_initial_search_pattern}${date}"
sudo zfs snapshot -r $rpool_snap 
sudo zfs send --raw -Rv $rpool_snap | gpg -r barl0g --encrypt -z 6 | ssh [email protected] "cat > $backup_path/${rpool_snap_name}_$(hostname)_$(date +%F).gpg"

When I do backup I typically do zfs snapshot -r rpool@YYYYMMDD-Some-Name. Or if you want ALL User Data you could do zfs snapshot -r rpool/USERDATA@YYYYMMDD-Some-Name instead.

But once again, you are asking us to debug blindly. We have NO IDEA if you did wrong in the Backup Process, or something is acting up during the Restore Process. We'd have to see Both Scripts to comment better.

Also no clue what is a "ZFS Image". Do you mean a "ZFS Snapshot" ? Which you backed up as zfs send -Rv rpool/USERDATA@YYYYMMDD-Some-Name | zfs receive -du somewhere/else/USERDATA@YYYYMMDD-Some-Name on another ZFS Pool (built from either a Physical VDEV or some .img File-based VDEV) ?

Yes. I save zfs send to a file.

"A Simple Archive" is also not very clear. How did you build that ? If you just did a tar or similar of /home, of course /root will NOT be included in that. Similarly, if for whatever Reason (like the Mounting order of ZFS Generators for instance) some Filesystem did NOT mount correctly, then your tar would also NOT contain the Backup of that relevant Mountpoint ...

I'm not sure if it matters in this case, but yes it is a simple tar archive with selected files (not all). This archive has everything it should have. He's fine.

I think the data loss problem has something to do with the wrong USERDATA/ mount point (as you wrote)

EDIT 1: Let me remind you that the initial problem has nothing to do with backup. It happened after a normal reboot. Although, I did perform an incremental backup (not restore) the day before. And now the new problem with the absence of /USERDATA/root is not critical for me. And again, this may be due to the wrong mount point.

luckylinux Oct 12, 2024

Well for USERDATA/user_a6ehn3 I don't know. As I said, you provided half Information "here" half "there". Like did the dataset Disappear AND the whole Pool (Root Dataset) size Shrink or the Total Pool size stayed the same and the Dataset Disappeared ? That's (also) not clear. If shrinking indeed the Dataset would be "gone", if not zsys might have done some very weird Stuff and merged it with / for whatever weird Reason.

Did

zfs list -t snapshot

Show anything at all ?

As to why Ubuntu zsys or other components of the ZFS System would do that I don't have a Clue to be honest. Very weird indeed. Did you by any chance have TRIM enabled (either autotrim or trimming manually), if SSD ? I remember there was some BUG about that (although I don't recall causing one entire Dataset to disappear just like that).

barl0g Oct 12, 2024
Author

Well for USERDATA/user_a6ehn3 I don't know. As I said, you provided half Information "here" half "there". Like did the dataset Disappear AND the whole Pool (Root Dataset) size Shrink or the Total Pool size stayed the same and the Dataset Disappeared ? That's (also) not clear. If shrinking indeed the Dataset would be "gone", if not zsys might have done some very weird Stuff and merged it with / for whatever weird Reason.

I, too, at first thought that the size of the entire pool remained the same and tried to find my data using ncdu, zfs list, zfs list -t snapshot, but then on #openzfs we did the calculation and found out that the pool size had decreased by about 50G. At first I incorrectly took into account the size of snapshots.

Did

zfs list -t snapshot

I can no longer do this on old system, but as I said, the snapshots of my data also disappeared. But they were because the -r option was used.

Show anything at all ?

Only docker snapshots and manual snapshots for all but not for USERDATA/root and USERDATA/user

As to why Ubuntu zsys or other components of the ZFS System would do that I don't have a Clue to be honest. Very weird indeed. Did you by any chance have TRIM enabled (either autotrim or trimming manually), if SSD ? I remember there was some BUG about that (although I don't recall causing one entire Dataset to disappear just like that).

TRIM enabled. The pool created with -o autotrim=on.

luckylinux Oct 12, 2024

I think I'm out of Ideas then 😞.

The Autotrim WAS an issue back then (at least in some Cases) but I'm not sure if your ZFS Version is "Recent enough".

As I said, I run OpenZFS (ZFS built from Source) on Ubuntu Desktops (and Rock 5B for aarch64, that's another Story though) since I heard that Ubuntu, at least at some Point, was doing some "Crazy Stuff" with ZFS.

As such, I came up with this Script for Debian-Based Distributions (it required a few Adjustments since ZFS 2.2.6).

I know that cannot help you recover the Data you lost (between the actual Data and the most recent Backup you took), but maybe that could be an Option to consider moving forward.

I also uninstalled zsys of course.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZFS userdata has disappeared without any errors after the reboot #16614

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

ZFS userdata has disappeared without any errors after the reboot #16614

barl0g Oct 7, 2024

Replies: 7 comments · 9 replies

amotin Oct 7, 2024 Collaborator

barl0g Oct 8, 2024 Author

IvanVolosyuk Oct 7, 2024

barl0g Oct 8, 2024 Author

ofthesun9 Oct 8, 2024

barl0g Oct 8, 2024 Author

barl0g Oct 8, 2024 Author

luckylinux Oct 9, 2024

barl0g Oct 9, 2024 Author

barl0g Oct 9, 2024 Author

barl0g Oct 9, 2024 Author

luckylinux Oct 10, 2024

barl0g Oct 10, 2024 Author

luckylinux Oct 12, 2024

barl0g Oct 12, 2024 Author

luckylinux Oct 12, 2024

barl0g
Oct 7, 2024

Replies: 7 comments 9 replies

amotin
Oct 7, 2024
Collaborator

barl0g Oct 8, 2024
Author

IvanVolosyuk
Oct 7, 2024

barl0g Oct 8, 2024
Author

ofthesun9
Oct 8, 2024

barl0g Oct 8, 2024
Author

barl0g
Oct 8, 2024
Author

luckylinux
Oct 9, 2024

barl0g Oct 9, 2024
Author

barl0g
Oct 9, 2024
Author

barl0g
Oct 9, 2024
Author

barl0g Oct 10, 2024
Author

barl0g Oct 12, 2024
Author