How are multiple datasets loaded?

[fxnie]

local_setup.yml：
"data_path": "/workspace/mnt/cm-nfx/gpt-neox/data/mamba/mamba-2.8b/algorithmic_corpus/algorithmic_corpus_text_document,/workspace/mnt/cm-nfx/gpt-neox/data/mamba/mamba-2.8b/opencode_sft/opencode_sft_text_document,/workspace/mnt/cm-nfx/gpt-neox/data/mamba/mamba-2.8b/synthetic_code_snippet/synthetic_code_snippet_text_document,/workspace/mnt/cm-nfx/gpt-neox/data/mamba/mamba-2.8b/synthetic_qa/synthetic_qa_text_document",

Another question is how to reduce the ratio of the test set and validation set to 0 when setting the training set ratio?

[iPRET]

Not sure if this helps, cause I myself am a user, not a maintainer, but:
I think a nice alternative to doing the data splits is to just define the training, validation and test datasets explicitly. Like e.g.:

{
  "train_data_paths": ["/scratch/project_465001281/IP/tokenized/gpt_neox/en3_text_document", "/scratch/project_465001281/IP/tokenized/gpt_neox/lv3_text_document"],
  "valid_data_paths": ["/scratch/project_465001281/IP/tokenized/gpt_neox/en3_flores_text_document"],
  "test_data_paths": ["/scratch/project_465001281/IP/tokenized/gpt_neox/en3_flores_text_document"]
}

Also in the example above it shows how one can load two datasets in parallel ("train_data_paths").

[fxnie]

How should I set up using only the training set and not the test and validation sets?

[iPRET]

A workaround could be to just generate some small dummy datasets, like e.g. from a .jsonl file with the line {"text": "eeeeeeeeee"}, and set eval_interval to a large number, if you really don't care about testing the model.

[fxnie]

Is there a way to load only the data from these two files as described by you? "/cramch/project-465001281/IP/tokenized/gpt_neox/en3_text.dideo" and "/cramch/project-465001281/IP/tokenized/gpt_neox/lv3_text.dideo"