"err": "creating file: creating ODSQ4 file: creating ODS file: creating ODS file: open /home/celestia-test/.celestia-bridge-mocha-4/blocks/AE7F681FC3379C028CD36885C28C3BF912DF54AE7F6210E4378C1A8029650BCA.ods: no space left on device"}

[mindstyle85]

Celestia Node version

v0.20.4

OS

Ubuntu

Install tools

No response

Others

No response

Steps to reproduce it

n/a

Expected result

no error

Actual result

so 3 people are getting these on Mocha:

2025-01-20T14:29:22.755Z ERROR header/sync sync/sync.go:227 syncing headers {"from": 4264271, "to": 4265030, "err": "creating file: creating ODSQ4 file: creating ODS file: creating ODS file: open /home/celestia-test/.celestia-bridge-mocha-4/blocks/AE7F681FC3379C028CD36885C28C3BF912DF54AE7F6210E4378C1A8029650BCA.ods: no space left on device"}

observations so far:

• one has ulimit set to 10m, currently it looks like his node has 5m open files
• plenty of disk space left
• its not the same block for them

Relevant log output

Is the node "stuck"? Has it stopped syncing?

No response

Notes

No response

[mindstyle85]

tagging @trader-payne

can you maybe also provide OS and filesystem you use?

[trader-payne]

OS - ubuntu 22.04.5
filesystem - ext4 (4 disks in raid0 via mdadm)

[murakamikaze]

Experienced the same:

Ubuntu 24.04
filesystem - ext4 (6 disks, raid0)

resyncinc with xfs now, will revert if that helps

[geldmon]

Ubuntu 22.04
filesystem - ext4 (2 disks, raid0)

[milakorn69]

OS - ubuntu 22.04.6
filesystem - ext4 (4 disks, raid0)

[Wondertan]

Ok, so the common thing is raid0. We need to research details about it and how it could be incompatible with node leading to this

[itrocket-team]

Original Server:

• Issue: Encountered an error: no space left on device
• Filesystem: ext4 with 2 disks - RAID 0
• Node Location: /home1/celbridge/.celestia-bridge-mocha-4/ /dev/md2
• OS Setup: Ubuntu 22.04.5 LTS is installed on a separate RAID 1 disk /dev/sda2

celbridge@host8085:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            76G  2.4M   76G   1% /run
efivarfs        304K  118K  182K  40% /sys/firmware/efi/efivars
/dev/sda2       219G   24G  185G  12% /
tmpfs           378G     0  378G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       511M  6.1M  505M   2% /boot/efi
/dev/md1         28T  3.0T   24T  12% /home
/dev/md2        2.9T  1.4T  1.4T  50% /home1
tmpfs            76G  4.0K   76G   1% /run/user/1001

Backup Server:

• Moved the node to this server because of issues on the original server. Everything works fine here.
• OS Version: Ubuntu 22.04.5 LTS.
• Filesystem: ext4 with 4 disks - RAID 0.
• Node and System Setup: Both the operating system and the node are located on the same RAID 0 disk.
• Node Location: /home/celbridge/.celestia-bridge-mocha-4/

celbridge@Ubuntu-2204-jammy-amd64-base:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            13G  1.2M   13G   1% /run
/dev/md2         14T  9.5T  3.7T  73% /
tmpfs            63G     0   63G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/md1        3.9G  429M  3.3G  12% /boot
tmpfs            13G  4.0K   13G   1% /run/user/1001

[itrocket-team]

Tried copying from /dev/md2 to /dev/md1, but encountered these errors.
sudo cp -r "/home1/celbridge/.celestia-bridge-mocha-4" /home/backup/

cp: cannot create regular file '/home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods': No space left on device

celbridge@host8085:~$ ls -l /home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods  
ls: cannot access '/home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods': Permission denied

celbridge@host8085:~$ sudo ls -l /home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods
ls: cannot access '/home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods': No such file or directory

celbridge@host8085:~$ stat /home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods
stat: cannot statx '/home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods': Permission denied

celbridge@host8085:~$ sudo stat /home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods
stat: cannot statx '/home/backup/celbridge/.celestia-bridge-mocha-4/blocks/3DDDB39873CF9CACD31E1A4E933F400B838F9474967A5BE13AA90EC8FC3C40A0.ods': No such file or directory

[murakamikaze]

Just resynced with xfs on the disk that holds the celestia bridge data, and I'm not encountering the previous issue.

Unclear if because resync or new filesystem.

Perhaps someone can try to resync without doing anything to further debug?

[antony-everstake]

We have the same issue with mocha-4 bridge.
Our server instance:

• Issue: Encountered an error on celestia-node: "err": "creating file: creating ODSQ4 file: creating Q4 file: creating Q4 file: open /celestia/celestia-bridge/blocks/3123D8B33856EBA7BE4E9D9629EC45C3F9E2F726BB4DEC3A91228EA4553468AA.q4: no space left on device"}

• Filesystem: ext4 with 4 disks with LVM and without any RAID

• Node Location: /celestia/celestia-bridge

• OS Setup: Ubuntu 24.04.1 LTS is installed on LVM partion

[SimonKraus]

You need to enable large directory features.

If you're having the data on a separate partition it's easy to just stop the node, unmount the partition and enable large_dir:
tune2fs -O large_dir /dev/sdbX
(replace sdbX with whatever partition used)

Then perform a filesystem consistency check
fsck -f /dev/sdbX

and mount the partition again.

[antony-everstake]

Thanks it is worked.

[Wondertan]

Ok, so the issue is not with raid0, but with ext4 having default cap on maximum amount of files allowed, that luckily can be increased as shown here.