-
Notifications
You must be signed in to change notification settings - Fork 63
Cray DataWarp Burst Buffer Support
It is possible to access datawarp burstbuffers from within a shifter container, however the setup needs to be done site-wide in your udiRoot.conf. The supported access to ensure that the burst buffers are bind-mounted within the container to allow data to be read and written. A user can then optionally volume-mount a burstbuffer to a mountpoint within their image.
In the siteFs section of udiRoot.conf, add the following as a new mount
/var/opt/cray/dws:/var/opt/cray/dws:rec:slave
ensure that you properly add line continuation and semicolon delimiters if there are other siteFs volume mounts in your configuration.
The "rec" flag is used to recursively bind-mount all the mount points (burst-buffers) under /var/opt/cray/dws into all shifter containers. The "slave" flag switches the bind mount from "private" mode, which is the default for Shifter bind mounts, to "slave" mode. In slave mode, any mount or unmount events from the external environment will be propagated into the container. This allows new burst-buffers to be added while the container is instantiated. This feature is important for the SLURM integration.
If you are using the shifter spank plugin to integrate with SLURM, the per-node prolog will run before the datawarp mounts are performed on the compute node. Thus, it is important to use the "slave" mount flag to remove the need for strict ordering of shifter setup relative to datawarp setup.
The "slave" mount flag does not work properly in some versions of Linux, and not at all in Cray CLE5.2. For those systems it is possible to work around this issue by using the "extern_setup" script to run the slurm_bb_support executable from the "extra" directory within the udiRoot source distribution.
Unfortunately, because the datawarp mount point is not known when shifter setup is running, it is at-present not possible to do a user-specified volume mount of a burstbuffer, meaning you cannot do something like:
salloc --image=docker:ubuntu:15.10 --volume='$DW_JOB_STRIPED:/input'
The above is also not possible because at present shifter cannot evaluate environment variables directly in volume mounts and relies on your job-submission shell to do so. Now that the extern_setup capability exists, we may be able to use that mechanism in the future to do some final operations like this, that will need to be evaluated for feasibility.