generate-51514137.err

  Running command git clone --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-hi4d643h/transformers_236632e49ec549719312411899822759
  Running command git clone --quiet https://github.com/huggingface/peft.git /tmp/pip-install-hi4d643h/peft_e80c6b4d36c04aaeb92d5e20783ca181
  Running command git clone --quiet https://github.com/huggingface/accelerate.git /tmp/pip-install-hi4d643h/accelerate_44fe9f616d2e4beebdbd99138273eb18
/home/brc4cb/.conda/envs/falcon_40B/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so.11.0'), PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)

Loading checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading checkpoint shards:   7%|▋         | 1/14 [00:04<00:52,  4.01s/it]
Loading checkpoint shards:  14%|█▍        | 2/14 [00:06<00:37,  3.12s/it]
Loading checkpoint shards:  21%|██▏       | 3/14 [00:09<00:32,  2.95s/it]
Loading checkpoint shards:  29%|██▊       | 4/14 [00:12<00:31,  3.15s/it]
Loading checkpoint shards:  36%|███▌      | 5/14 [00:15<00:27,  3.01s/it]
Loading checkpoint shards:  43%|████▎     | 6/14 [00:18<00:22,  2.86s/it]
Loading checkpoint shards:  50%|█████     | 7/14 [00:20<00:19,  2.73s/it]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:23<00:16,  2.78s/it]
Loading checkpoint shards:  64%|██████▍   | 9/14 [00:26<00:15,  3.01s/it]
Loading checkpoint shards:  71%|███████▏  | 10/14 [00:29<00:11,  2.85s/it]
Loading checkpoint shards:  79%|███████▊  | 11/14 [00:32<00:09,  3.03s/it]
Loading checkpoint shards:  86%|████████▌ | 12/14 [00:35<00:06,  3.05s/it]
Loading checkpoint shards:  93%|█████████▎| 13/14 [00:38<00:02,  2.95s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [00:39<00:00,  2.37s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [00:39<00:00,  2.83s/it]
slurmstepd: error: *** JOB 51514137 ON udc-an36-19 CANCELLED AT 2023-07-10T19:02:15 DUE TO TIME LIMIT ***