Unable to create tables disk size

[drgroot]

I have currently deployed a starrocks cluster with 1 FE and 6 BE nodes in a share nothing topology in k8s.

I am using starrocks/fe-ubuntu, starrocks/be-ubuntu and tag 3.3-latest

Problem
When following the guide to load data via insert, I get an error

query:

CREATE DATABASE IF NOT EXISTS load_test;
USE load_test;
CREATE TABLE insert_wiki_edit
(
    event_time      DATETIME,
    channel         VARCHAR(32)      DEFAULT '',
    user            VARCHAR(128)     DEFAULT '',
    is_anonymous    TINYINT          DEFAULT '0',
    is_minor        TINYINT          DEFAULT '0',
    is_new          TINYINT          DEFAULT '0',
    is_robot        TINYINT          DEFAULT '0',
    is_unpatrolled  TINYINT          DEFAULT '0',
    delta           INT              DEFAULT '0',
    added           INT              DEFAULT '0',
    deleted         INT              DEFAULT '0'
)
DUPLICATE KEY(
    event_time,
    channel,
    user,
    is_anonymous,
    is_minor,
    is_new,
    is_robot,
    is_unpatrolled
)
PARTITION BY RANGE(event_time)(
    PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'),
    PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'),
    PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'),
    PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00')
)
DISTRIBUTED BY HASH(user);

CREATE TABLE source_wiki_edit
(
    event_time      DATETIME,
    channel         VARCHAR(32)      DEFAULT '',
    user            VARCHAR(128)     DEFAULT '',
    is_anonymous    TINYINT          DEFAULT '0',
    is_minor        TINYINT          DEFAULT '0',
    is_new          TINYINT          DEFAULT '0',
    is_robot        TINYINT          DEFAULT '0',
    is_unpatrolled  TINYINT          DEFAULT '0',
    delta           INT              DEFAULT '0',
    added           INT              DEFAULT '0',
    deleted         INT              DEFAULT '0'
)
DUPLICATE KEY(
    event_time,
    channel,user,
    is_anonymous,
    is_minor,
    is_new,
    is_robot,
    is_unpatrolled
)
PARTITION BY RANGE(event_time)(
    PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'),
    PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'),
    PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'),
    PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00')
)
DISTRIBUTED BY HASH(user);

] failed to find any backend with qualified disk usage from 5 candidate backends, needAvailable=true, 

Expected
I can create the table and insert data

Toubleshooting

1. I notice the be nodes are healthy, but reporting NO STORAGE

mysql> show backends;
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+
| BackendId | IP                                                      | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version       | Status                                 | DataTotalCapacity | DataUsedPct | CpuCores | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics | Location |
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+
| 10001     | test-be-0.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:12 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10002     | test-be-1.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:37 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10003     | test-be-2.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:47 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10042     | test-be-4.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:29:52 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10047     | test-be-5.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:30:07 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+

2. The storage_root_path is properly set in the /opt/starrocks/be/conf/be.conf location and being set properly when running a curl on the variables

$ curl http://test-be-0.test-be.svc.cluster.local:8040/varz | grep storage_root_path
storage_root_path=/opt/starrocks/be/storage

3. The storage/disk is mounted correctly and there is a good amount of space

# df -h | grep storage; exit
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd        9800G  6.3M  9800G   1% /opt/starrocks/be/storage

Any thoughts?