Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hydra's cascading AttributeErrors #3

Open
nuankw opened this issue Nov 22, 2023 · 3 comments
Open

Hydra's cascading AttributeErrors #3

nuankw opened this issue Nov 22, 2023 · 3 comments

Comments

@nuankw
Copy link

nuankw commented Nov 22, 2023 •
edited
Loading

Hello Authors,

We are trying to reproduce the micro model on 2 GPUs and run into a cascade of AttributeErrors as pasted below. We wonder if you could provide us some pointers on how to fix this issue. Many thanks!

(flenv) [nuanwen@d14-03 training]$ torchrun --nproc_per_node=2 run.py  experiment=owt/backpack-micro-flash     trainer.devices=2    name=backpack-micro-flash-fp16     datamodule.batch_size=128

...


[2023-11-21 19:44:28,586][lightning_lite.utilities.seed][INFO] - [rank: 0] Global seed set to 1111
[2023-11-21 19:44:28,617][src.tasks.seq][INFO] - Instantiating model <src.models.backpack.BackpackLMHeadModel>
Error executing job with overrides: ['experiment=owt/backpack-micro-flash', 'trainer.devices=2', 'name=backpack-micro-flash-fp16', 'datamodule.batch_size=128']
Error executing job with overrides: ['experiment=owt/backpack-micro-flash', 'trainer.devices=2', 'name=backpack-micro-flash-fp16', 'datamodule.batch_size=128']
Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 644, in _locate
Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 644, in _locate
    obj = getattr(obj, part)
AttributeError: module 'src.models' has no attribute 'backpack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 650, in _locate
    obj = getattr(obj, part)
AttributeError: module 'src.models' has no attribute 'backpack'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 650, in _locate
    obj = import_module(mod)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
    obj = import_module(mod)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked                                                                                                                                                   
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/models/backpack.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/models/backpack.py", line 19, in <module>
    from flash_attn.modules.mlp import Mlp, FusedDenseGeluDense, ParallelFusedDenseGeluDense
ImportError: cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
    from flash_attn.modules.mlp import Mlp, FusedDenseGeluDense, ParallelFusedDenseGeluDense
ImportError: cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
    target = _locate(target)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 658, in _locate
        target = _locate(target)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 658, in _locate
raise ImportError(    
ImportError: Error loading 'src.models.backpack.BackpackLMHeadModel':
ImportError("cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
raise ImportError(    
ImportError: Error loading 'src.models.backpack.BackpackLMHeadModel':
ImportError("cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target                                                                          
return _target_(*args, **kwargs)    
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 33, in __init__
return _target_(*args, **kwargs)
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 33, in __init__
    self.instantiate_model()
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 56, in instantiate_model
    self.instantiate_model()
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 56, in instantiate_model
    self.model = hydra.utils.instantiate(self.model_cfg, _recursive_=recursive)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    self.model = hydra.utils.instantiate(self.model_cfg, _recursive_=recursive)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
    return instantiate_node(
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
    _target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
_target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
        raise InstantiationException(msg) from eraise InstantiationException(msg) from e
hydra.errors
.InstantiationExceptionhydra.errors.: Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: cfg.modelInstantiationException

The above exception was the direct cause of the following exception:

: Traceback (most recent call last):
Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: cfg.model
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 68, in <module>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 68, in <module>
    main()
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
        main()
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_hydra(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
_run_app(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
run_and_report(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
raise ex    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
return func()    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
lambda: hydra.run(    
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    _ = ret.return_value
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    raise self._return_value
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
        ret.return_value = task_function(task_cfg)ret.return_value = task_function(task_cfg)

  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 62, in main
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 62, in main
    return train(config)
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/train.py", line 50, in train
    return train(config)
  File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/train.py", line 50, in train
    model: LightningModule = hydra.utils.instantiate(config.task, cfg=config, _recursive_=False)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    model: LightningModule = hydra.utils.instantiate(config.task, cfg=config, _recursive_=False)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return instantiate_node(
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.tasks.seq.SequenceLMModel':
InstantiationException("Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.\nfull_key: cfg.model")
full_key: task
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.tasks.seq.SequenceLMModel':
InstantiationException("Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.\nfull_key: cfg.model")
full_key: task
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 46950) of binary: /home1/nuanwen/.conda/envs/flenv/bin/python
Traceback (most recent call last):
  File "/home1/nuanwen/.conda/envs/flenv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
run.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-11-21_19:44:32
  host      : d14-03.hpc.usc.edu
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 46951)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-11-21_19:44:32
  host      : d14-03.hpc.usc.edu
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 46950)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
@john-hewitt
Copy link
Owner

Hey, thanks for the ping.

The error out of those that seems to cause the others is

ImportError: cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)

which suggests to me that the kernels from NVIDIA like FusedDenseGeluDense are not installed; instructions for installing them are here.

@nuankw
Copy link
Author

nuankw commented Nov 23, 2023

Hi John, thank you so much for your quick reply and pointing out the source of the problem! We looked into it and realized that the issue is caused by installing flash_attn via pip install flash_attn after failing at pip install --verbose -e . . (Error log attached below)

We attempted to run the latter command on different GPU configurations (A100, A6000 and V100) but this line of command seems not working on all of them. Do you mind telling us a bit more about your A100 settings?

(flenv) [nuanwen@a11-01 backpacks-flash-attn]$ ninja --version
1.11.1.git.kitware.jobserver-1
(flenv) [nuanwen@a11-01 backpacks-flash-attn]$ echo $?
0
(flenv) [nuanwen@a11-01 backpacks-flash-attn]$ pip install --verbose -e . 
Using pip 23.3 from /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/pip (python 3.9)
Obtaining file:///project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn
  Running command python setup.py egg_info
  error: pathspec 'csrc/flash_attn/cutlass' did not match any file(s) known to git
  /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/dist.py:509: InformationOnly: Normalizing '0.2.6-1' to '0.2.6.post1'
    self.metadata.version = self._normalize_version(


  torch.__version__  = 1.13.1+cu117


  running egg_info
  creating /tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info
  writing /tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.cu' under directory 'flash_attn'
  warning: no files found matching '*.h' under directory 'flash_attn'
  warning: no files found matching '*.cuh' under directory 'flash_attn'
  warning: no files found matching '*.cpp' under directory 'flash_attn'
  adding license file 'LICENSE'
  adding license file 'AUTHORS'
  writing manifest file '/tmp/pip-pip-egg-info-38csupox/flash_attn.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages (from flash-attn==0.2.6.post1) (1.13.1+cu117)
Requirement already satisfied: einops in /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages (from flash-attn==0.2.6.post1) (0.7.0)
Requirement already satisfied: typing-extensions in /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages (from torch->flash-attn==0.2.6.post1) (4.8.0)
DEPRECATION: pytorch-lightning 1.8.1 has a non-standard dependency specifier torch>=1.9.*. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: flash-attn
  Attempting uninstall: flash-attn
    Found existing installation: flash-attn 2.3.4
    Uninstalling flash-attn-2.3.4:
      Removing file or directory /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn-2.3.4.dist-info/
      Removing file or directory /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/
      Removing file or directory /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn_2_cuda.cpython-39-x86_64-linux-gnu.so
      Successfully uninstalled flash-attn-2.3.4
  Running setup.py develop for flash-attn
    Running command python setup.py develop
    error: pathspec 'csrc/flash_attn/cutlass' did not match any file(s) known to git
    /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/dist.py:509: InformationOnly: Normalizing '0.2.6-1' to '0.2.6.post1'
      self.metadata.version = self._normalize_version(


    torch.__version__  = 1.13.1+cu117


    running develop
    /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)
    /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` directly.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
            ********************************************************************************

    !!
      self.initialize_options()
    running egg_info
    writing flash_attn.egg-info/PKG-INFO
    writing dependency_links to flash_attn.egg-info/dependency_links.txt
    writing requirements to flash_attn.egg-info/requires.txt
    writing top-level names to flash_attn.egg-info/top_level.txt
    reading manifest file 'flash_attn.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.cu' under directory 'flash_attn'
    warning: no files found matching '*.h' under directory 'flash_attn'
    warning: no files found matching '*.cuh' under directory 'flash_attn'
    warning: no files found matching '*.cpp' under directory 'flash_attn'
    adding license file 'LICENSE'
    adding license file 'AUTHORS'
    writing manifest file 'flash_attn.egg-info/SOURCES.txt'
    running build_ext
    building 'flash_attn_cuda' extension
    Emitting ninja build file /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    [1/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.cu:29:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.cu:29:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_fprop_fp16_kernel.sm80.cu:29:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [2/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim32.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim32.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim32.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [3/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_block_dgrad_fp16_kernel_loop.sm80.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [4/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim64.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim64.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim64.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [5/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim128.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim128.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim128.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [6/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim128.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim128.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim128.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim128.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [7/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim64.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim64.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_fwd_hdim64.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:10,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_launch_template.h:12,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fwd_hdim64.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    [8/8] /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim32.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    FAILED: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim32.o
    /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/bin/nvcc  -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src -I/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/cutlass/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/TH -I/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/include/THC -I/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/cuda-11.7.1-jamqdjs/include -I/home1/nuanwen/.conda/envs/flenv/include/python3.9 -c -c /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu -o /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/build/temp.linux-x86_64-cpython-39/csrc/flash_attn/src/fmha_bwd_hdim32.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=flash_attn_cuda -D_GLIBCXX_USE_CXX11_ABI=0
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:5,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/static_switch.h:28:40: warning: backslash-newline at end of file
       28 | #define FP16_SWITCH(COND, CODE)        \
          |
    In file included from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/smem_tile.h:32,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_kernel.h:34,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_fprop_kernel_1xN.h:31,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_dgrad_kernel_1xN_loop.h:6,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_launch_template.h:7,
                     from /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha_bwd_hdim32.cu:5:
    /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/csrc/flash_attn/src/fmha/gemm.h:32:10: fatal error: cutlass/cutlass.h: No such file or directory
       32 | #include "cutlass/cutlass.h"
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
        subprocess.run(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/subprocess.py", line 524, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/setup.py", line 157, in <module>
        setup(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
        return distutils.core.setup(**attrs)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
        super().run_command(command)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/command/develop.py", line 111, in install_for_development
        self.run_command('build_ext')
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
        self.distribution.run_command(command)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
        super().run_command(command)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 84, in run
        _build_ext.run(self)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
        self.build_extensions()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 843, in build_extensions
        build_ext.build_extensions(self)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
        self._build_extensions_serial()
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
        self.build_extension(ext)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
        _build_ext.build_extension(self, ext)
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
        objects = self.compiler.compile(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 658, in unix_wrap_ninja_compile
        _write_ninja_file_and_compile_objects(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1573, in _write_ninja_file_and_compile_objects
        _run_ninja_build(
      File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /home1/nuanwen/.conda/envs/flenv/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize
    
    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)
    
    __file__ = %r
    sys.argv[0] = __file__
    
    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"
    
    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/
  Rolling back uninstall of flash-attn
  Moving to /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn-2.3.4.dist-info/
   from /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/~lash_attn-2.3.4.dist-info
  Moving to /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/
   from /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/~lash_attn
  Moving to /home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn_2_cuda.cpython-39-x86_64-linux-gnu.so
   from /tmp/pip-uninstall-sv09_dwd/flash_attn_2_cuda.cpython-39-x86_64-linux-gnu.so
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

@john-hewitt
Copy link
Owner

Using pip install flash_attn doesn't work because this codebase was written with an earlier development version of flash_attn (i.e., the version that this codebase forked at.) So I'd get rid of any existing flash_attn install.

It looks like you're missing cutlass, an NVIDIA C library that's included via a git submodule:

backpacks-flash-attn/.gitmodules

Line 2 in d60371d

path = csrc/flash_attn/cutlass

I think this would be initialized during python setup.py install in the root directory of this repository (as here).

Not sure why this didn't seem necessary in my GCP replication... Let me know how it goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@john-hewitt @nuankw