What space should be used for the IWAA calibration/regionalization work?

[arezoorn]

As a part of the new project, I would be doing calibration at selected small basins over the CONUS and later we would do long term model runs at CONUS scale. Where is the space that I could use for this work on Denali?

[rsignell-usgs]

@pnorton-usgs:

• We should use /caldera/hytest_scratch/scratch/ right?
• How much space do we have there?
• Does @arezoorn need to be added to a specific group to write to that?

[pnorton-usgs]

@rsignell-usgs I'm not sure if hytest_scratch is the best place for running calibrations and simulations on. I know some of the solid state volumes are read/write rate-limited to optimize drive life but I'm not sure about hytest_scratch. I will check with Jeff on this.

Currently there is about 328 TB available, however, the conus404 bias-adjusted dataset will need about 80 TB which reduces the total available to about 248 TB. Do we have any estimate of the amount of space needed for this work?

I think any user can write to /caldera/hytest_scratch/scratch, @arezoorn could write a small test file to verify this. If it doesn't work then they will need to be added to the hytest group.

[arezoorn]

@pnorton-usgs I can get a rough estimate of how much space we would need. Due to the high number of files, it takes a long time to get the total size of the files that I have, therefore, I will get back to you with better estimates later. I would need space for domains cutout which is pretty small (few GB), forcing cutouts for all basins (several TB), model calibration (several TB), and CONUS scale model runs which is usually the largest (at least 50 TB).

The calibration part is IO intensive, so your concern is valid. Thank you for looking into it!

I will check whether I have access to scratch space or not, and come back to you on that.

[pnorton-usgs]

I got a response back -- there is no rate limit on read/write for hytest_scratch. Peak bandwidth is 21 GB/s which is quite a bit slower than our Lustre project area (36 GB/s) but the project area has quotas which is about 50 TB per user.

[arezoorn]

Confirming that I could access the /caldera/hytest_scratch/scratch space.

Regarding the 50TB quota, could that be changed? The bias corrected data is 80 TB, and quota of one of us would be filled with just moving that dataset. Or maybe I am missing something .

[pnorton-usgs]

The /caldera/hytest_scratch filesystem does not enforce quotas; it's the /home and /caldera/projects filesystems that do enforce quotas.

[arezoorn]

Thanks for clarification on this! It was my confusion. We will be moving data very soon to the space.

[arezoorn]

Does anyone know what account should the IWAA calibration work be charged to? I saw the below in one of my colleagues s slurm submission script, and I wanted to see whether I could use that account or there is a different account setup.
#SBATCH --account=impd
Thanks!