Migrate files on Lustre between OSTs.
The file migration is done from source to target OSTs until a specific fill threshold is reached.
To keep the load of the Lustre OSS low, the file migration does a 1-to-1 mapping between source to target OSTs.
If more than one source and target OST should be used to increase the file migration processes at a time,
this feature must be implemented (see issue).
To migrate files on Lustre OSTs Cyclone uses so called input files.
An input file consists of datasets with two columns separated with a whitespace for each file to migrate:
- Decimal OST index
- Filepath
The OST indexes within input files define the source OST indexes that Cyclone will migrate data from.
Therefore it is not required to specify those indexes, since Cyclone will determine them automatically.
To speed up a migration, Robinhood is used to determine the OST index where a file is located on Lustre.
Please refer to the following slides for a Lustre file migration.
As for the task result description please refer to the lfs-utils library MigrateResult class.
Example config file for the task generator
| Name | Type | Value | Description |
|---|---|---|---|
| local_mode | String | yes/no, on/off, true/false and 1/0 | Specifies if local or productive mode is enabled |
| Name | Type | Value | Description |
|---|---|---|---|
| num_osts | Int | 1-1000 | Specifies the number of Lustre OSTs to simulate |
| Name | Type | Value | Description |
|---|---|---|---|
| update_fill_level | Int | 1-3600 | Time in seconds when to update Lustre OSTs fill level |
| reload_files | Int | 1-3600 | Time in seconds when to reload input files |
| print_caches | Int | 1-3600 | Time in seconds when to print the caches with number of files to migrate |
| Name | Type | Value | Description |
|---|---|---|---|
| task_file | String | Path | Path to task config file |
| task_name | String | Name | Name of task to load |
| Name | Type | Value | Description |
|---|---|---|---|
| input_dir | String | Path | Path to input directory where to process input file lists from |
| ost_fill_threshold_source | Int | 0-90 | Lustre OST fill level threshold in percentage for reducing down source OSTs |
| ost_fill_threshold_target | Int | 1-90 | Lustre OST fill level threshold in percentage for filling up target OSTs |
| ost_targets | RangeSet | n>=0 | List of decimal OST indexes comma separated and ranges defined with hyphen |
| Name | Type | Value | Description |
|---|---|---|---|
| fs_path | String | Path | Lustre filesystem path |
Example config file for the task
| Name | Type | Value | Description |
|---|---|---|---|
| filename | String | - | Placeholder, filled during runtime |
| source_ost | Int | - | Placeholder, filled during runtime |
| target_ost | Int | - | Placeholder, filled during runtime |
| direct_io | Bool | True/False | Enables direct IO |
| block | Bool | True/False | Enables blocking file migration |
| skip | Bool | True/False | Skip stripped files |