generate-51603669.err

  Running command git clone --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-v8ju7rkv/transformers_b78197277f0b431fa6854977a2b0717f
  Running command git clone --quiet https://github.com/huggingface/peft.git /tmp/pip-install-v8ju7rkv/peft_2380e81a887740eeb26b0b4632f136bc
  Running command git clone --quiet https://github.com/huggingface/accelerate.git /tmp/pip-install-v8ju7rkv/accelerate_343eae11128645f0907dffa3388ddb80
/home/brc4cb/.conda/envs/falcon_40B/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so'), PosixPath('/home/brc4cb/.conda/envs/falcon_40B/lib/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)

Loading checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading checkpoint shards:   7%|▋         | 1/14 [00:04<00:59,  4.61s/it]
Loading checkpoint shards:  14%|█▍        | 2/14 [00:08<00:46,  3.90s/it]
Loading checkpoint shards:  21%|██▏       | 3/14 [00:11<00:41,  3.75s/it]
Loading checkpoint shards:  29%|██▊       | 4/14 [00:15<00:37,  3.72s/it]
Loading checkpoint shards:  36%|███▌      | 5/14 [00:20<00:37,  4.16s/it]
Loading checkpoint shards:  43%|████▎     | 6/14 [00:25<00:36,  4.51s/it]
Loading checkpoint shards:  50%|█████     | 7/14 [00:30<00:33,  4.73s/it]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:35<00:29,  4.90s/it]
Loading checkpoint shards:  64%|██████▍   | 9/14 [00:42<00:26,  5.38s/it]
Loading checkpoint shards:  71%|███████▏  | 10/14 [00:48<00:22,  5.72s/it]
Loading checkpoint shards:  79%|███████▊  | 11/14 [00:56<00:19,  6.42s/it]
Loading checkpoint shards:  86%|████████▌ | 12/14 [01:04<00:13,  6.89s/it]
Loading checkpoint shards:  93%|█████████▎| 13/14 [01:12<00:07,  7.22s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [01:15<00:00,  6.00s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [01:15<00:00,  5.42s/it]
slurmstepd: error: *** JOB 51603669 ON udc-an36-19 CANCELLED AT 2023-07-17T10:20:43 ***