generate-51616307.err

  Running command git clone --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-hjisyx51/transformers_94e36513a2ac4db6808ccfbbc0111501
  Running command git clone --quiet https://github.com/huggingface/peft.git /tmp/pip-install-hjisyx51/peft_5d4bc7937f244ead9d1b1e413c960729
  Running command git clone --quiet https://github.com/huggingface/accelerate.git /tmp/pip-install-hjisyx51/accelerate_4e4a97323bfb4e24b392f1c481f4e0ac
/home/brc4cb/.conda/envs/falcon_40B/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so'), PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)

Loading checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading checkpoint shards:   7%|▋         | 1/14 [00:06<01:22,  6.37s/it]
Loading checkpoint shards:  14%|█▍        | 2/14 [00:12<01:15,  6.26s/it]
Loading checkpoint shards:  21%|██▏       | 3/14 [00:19<01:12,  6.57s/it]
Loading checkpoint shards:  29%|██▊       | 4/14 [00:26<01:07,  6.71s/it]
Loading checkpoint shards:  36%|███▌      | 5/14 [00:33<01:00,  6.74s/it]
Loading checkpoint shards:  43%|████▎     | 6/14 [00:38<00:49,  6.18s/it]
Loading checkpoint shards:  50%|█████     | 7/14 [00:43<00:41,  5.96s/it]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:49<00:34,  5.80s/it]
Loading checkpoint shards:  64%|██████▍   | 9/14 [00:54<00:28,  5.72s/it]
Loading checkpoint shards:  71%|███████▏  | 10/14 [01:00<00:22,  5.57s/it]
Loading checkpoint shards:  79%|███████▊  | 11/14 [01:05<00:16,  5.58s/it]
Loading checkpoint shards:  86%|████████▌ | 12/14 [01:10<00:10,  5.47s/it]
Loading checkpoint shards:  93%|█████████▎| 13/14 [01:16<00:05,  5.51s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [01:19<00:00,  4.70s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [01:19<00:00,  5.67s/it]
slurmstepd: error: *** JOB 51616307 ON udc-an37-1 CANCELLED AT 2023-07-17T11:24:21 ***