Running out of diskspace has disastrous consequences

[zander]

Qubes OS version:

4.0 RC3, updated to 'testing' latest.

---

Steps to reproduce the behavior:

I was doing a system update in my archlinux template.
Apparently I ran out of disk space half way through.
While waiting for the download I was still working on the system and did things like starting some disposable vms to see how they behaved. Those started to fail starting and stopping.

At one point my graphical system completely froze and switching to a second tty I noticed that qubesd was frozen. Asking systemctl to restart it took various minutes.
In the end I had no choice but to hit the hardware reset button.

---

I started my archlinux template to assess the damage. I got lots of errors from ldconfig about empty files from software I just upgraded. I conclude that while I wasn't looking at the window, the upgrade finished successfully but the files were not actually there.
While trying to salvage this mess I reinstalled some large packages. The system hang again, I rebooted again.

At this point I realized this was actually about the drive-space.

---

When I got back in my desktop I noticed sys-net didn't even start.

The commandline was greeted with replies such as;

$ qvm-start fedora-25
b'  Cannot create new thin volume, free space in thin qubes_dom0/pool0 reached threshold.\n'

and;

$ qvm-start archlinux
volume qubes-dom0/vm-archlinux-private missing

Expected behavior:

I expect that disk-quotas set in the qubes system are honoured by hard limiting clients space usage, and I expect that there is no way to fill up any pool to the point where you can't even start sys-net.
I surely don't expect a private partition to go 'missing' due to this...

Actual behavior:

The filesystem inside of a Qube doesn't seem to be told about being out of diskspace. The system seems to adjust until it breaks. And it breaks spectacularly.

Loss of data, system looks unrecoverable and in need of reinstall.

General notes:

This was a test setup on a spare ssd, nothing was lost, just time.

[andrewdavidwong]

This appears to be a duplicate of #1872 + #2016. If you believe this is not really a duplicate, please leave a comment briefly explaining why. We'll be happy to take another look and, if appropriate, reopen this issue. Thank you.