Missing os deps when installing ReFrame on graviton nodes in our AWS environment

[casparvl]

[casparvl@fair-mastodon-c6g-2xlarge-0001 ~]$ rm -rf /tmp/reframe_421
[casparvl@fair-mastodon-c6g-2xlarge-0001 ~]$ python3 -m venv /tmp/reframe_421
[casparvl@fair-mastodon-c6g-2xlarge-0001 ~]$ source /tmp/reframe_421/bin/activate
(reframe_421) [casparvl@fair-mastodon-c6g-2xlarge-0001 ~]$ python3 -m pip install reframe-hpc==4.2.1
Collecting reframe-hpc==4.2.1
  Using cached https://files.pythonhosted.org/packages/aa/6c/a7a5cc465ef223bf686717b6034a817be7028f30b28b4d2b840f1c29bda3/ReFrame_HPC-4.2.1-py3-none-any.whl
Collecting archspec (from reframe-hpc==4.2.1)
  Using cached https://files.pythonhosted.org/packages/63/ae/333e7d216dda9134558ddc30792d96bfc58968ff5cc69b4ad9e02dfac654/archspec-0.2.1-py3-none-any.whl
Collecting argcomplete (from reframe-hpc==4.2.1)
  Using cached https://files.pythonhosted.org/packages/4f/ef/8b604222ba5e5190e25851aa3a5b754f2002361dc62a258a8e9f13e866f4/argcomplete-3.1.1-py3-none-any.whl
Collecting jsonschema (from reframe-hpc==4.2.1)
  Using cached https://files.pythonhosted.org/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl
Collecting PyYAML (from reframe-hpc==4.2.1)
  Using cached https://files.pythonhosted.org/packages/36/2b/61d51a2c4f25ef062ae3f74576b01638bebad5e045f747ff12643df63844/PyYAML-6.0.tar.gz
Collecting lxml (from reframe-hpc==4.2.1)
  Using cached https://files.pythonhosted.org/packages/06/5a/e11cad7b79f2cf3dd2ff8f81fa8ca667e7591d3d8451768589996b65dec1/lxml-4.9.2.tar.gz
    Complete output from command python setup.py egg_info:
    Building lxml version 4.9.2.
    Building without Cython.
    Error: Please make sure the libxml2 and libxslt development packages are installed.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-bospx5ra/lxml/
You are using pip version 9.0.3, however version 23.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@boegel can you have a look at this? It seems to be missing the development headers for libxml2 and libxslt.

[casparvl]

Just logging here some stuff from a chat conversation with @ocaisa :

Alan tried to follow https://cluster-in-the-cloud.readthedocs.io/en/latest/node-images.html , and the step to actually build the images didn't work. It claimed the AMI is not available. At that point, we backed off, as we didn't want to ruin the CITC environment by accident. We'll wait for @boegel to return and look at this

[casparvl]

Workaround: use the ReFrame from EESSI to manually generate the topology file. E.g.

srun -p aarch64-neoverse-n1-node --exclusive -t 1:00:00 --pty /bin/bash
source /cvmfs/pilot.eessi-hpc.org/versions/2023.06/init/bash
mkdir -p ~/.reframe/topology/Magic_Castle-aarch64-neoverse-N1-16c-32gb
reframe --detect-host-topology=~/.reframe/topology/Magic_Castle-aarch64-neoverse-N1-16c-32gb/processor.json

Once the topology files have been created, any subsequent run from ReFrame will pick up on it.

[casparvl]

If we can convince the CPU autodetect script to first upgrade pip, that would also work. See reframe-hpc/reframe#3023