Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2024-2.3 envs #41

Merged
merged 9 commits into from
Jul 31, 2024
Merged

2024-2.3 envs #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3b55ad2
2024-2.3 envs
mrakitin Jul 17, 2024
f92502b
Add `cookiecutter` to the list of packages
mrakitin Jul 17, 2024
8c7ac15
Add `julia`
mrakitin Jul 18, 2024
fcf1198
Add `pycryptodome` - dependency of nslsii -> sync-experiment
mrakitin Jul 19, 2024
5b0200c
Add `diffpy.pdfgui`
mrakitin Jul 23, 2024
6fb8b0f
Skip `diffpy.pdfgui` for py312
mrakitin Jul 24, 2024
87ebf43
Comment out pdfgui for py311
mrakitin Jul 25, 2024
1ad38d5
Return pdfgui to py311
mrakitin Jul 31, 2024
c51fa20
Move some packages to a separate installation step
mrakitin Jul 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,39 @@ jobs:
conda config --show
printenv | sort

- name: Add packages for py311
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice workaround! Glad it worked.

if: matrix.python-version == '3.11'
run: |
conda install -c conda-forge \
"bloptools>=0.7.0" \
"bluesky-darkframes>=0.6.0" \
caproto \
emojis \
happi \
pexpect \
"pyolog>=4.5.0" \
pyserial \
python-confluent-kafka \
pyzenodo3 \
simple-pid \
slack-sdk \
hklpy \
"hxnfly>=0.0.11" \
kkcalc \
ppmac \
"pychx>=4.3.1" \
"xpdacq==1.0.0" \
hunter \
logging_tree \
line_profiler \
pyinstrument \
pyperformance \
botorch \
gpytorch \
ortools-python \
pytorch \
scikit-optimize

- name: Export files
run: |
set -vxeo pipefail
Expand Down
4 changes: 2 additions & 2 deletions configs/config-py310.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
docker_image: "quay.io/condaforge/linux-anvil-cos7-x86_64:latest"
env_name: "2024-2.2-py310-tiled"
env_name: "2024-2.3-py310-tiled"
conda_env_file: "env-py310.yml"
conda_binary: "mamba"
python_version: "3.10"
Expand All @@ -19,7 +19,7 @@ zenodo_metadata:
title: "NSLS-II collection conda environment"
upload_type: "software"
description: "NSLS-II collection conda environment"
version: 2024-2.2-tiled
version: 2024-2.3-tiled
creators:
- name: Rakitin, Max
affiliation: "Brookhaven National Laboratory"
Expand Down
4 changes: 2 additions & 2 deletions configs/config-py311.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
docker_image: "quay.io/condaforge/linux-anvil-cos7-x86_64:latest"
env_name: "2024-2.2-py311-tiled"
env_name: "2024-2.3-py311-tiled"
conda_env_file: "env-py311.yml"
conda_binary: "mamba"
python_version: "3.11"
Expand All @@ -19,7 +19,7 @@ zenodo_metadata:
title: "NSLS-II collection conda environment"
upload_type: "software"
description: "NSLS-II collection conda environment"
version: 2024-2.2-tiled
version: 2024-2.3-tiled
creators:
- name: Rakitin, Max
affiliation: "Brookhaven National Laboratory"
Expand Down
4 changes: 2 additions & 2 deletions configs/config-py312.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
docker_image: "quay.io/condaforge/linux-anvil-cos7-x86_64:latest"
env_name: "2024-2.2-py312-tiled"
env_name: "2024-2.3-py312-tiled"
conda_env_file: "env-py312.yml"
conda_binary: "mamba"
python_version: "3.12"
Expand All @@ -19,7 +19,7 @@ zenodo_metadata:
title: "NSLS-II collection conda environment"
upload_type: "software"
description: "NSLS-II collection conda environment"
version: 2024-2.2-tiled
version: 2024-2.3-tiled
creators:
- name: Rakitin, Max
affiliation: "Brookhaven National Laboratory"
Expand Down
8 changes: 6 additions & 2 deletions envs/env-py310.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: 2024-2.2-py310-tiled
name: 2024-2.3-py310-tiled
channels:
- conda-forge
dependencies:
Expand All @@ -7,15 +7,20 @@ dependencies:
- bloptools >=0.7.0
- blosc-hdf5-plugin
- conftrak >=0.0.9
- cookiecutter
- dask >=2023.9.0
- distributed >=2023.9.0
- databroker >=2.0.0b45
- diffpy.pdffit2
- diffpy.pdfgui
- diffpy.structure
- event-model>=1.21
- hdf5-external-filter-plugins
- julia
- maggma >=0.66
- mp-api >=0.41.2
- pychx >=4.3.1
- pycryptodome
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I am glad pycryptodome is added in env-py3{10,11,12}.yml I have a general question about this type of packages.

It seems pycryptodome is already a dependency in nslsii (https://github.com/NSLS-II/nslsii/blob/2a31a9950d1fc266ae212c5c64ab6f83bab447b0/requirements.txt#L18) but for whatever reason the conda env is not resolving to that version of nslsii, so not picking up pycryptodome. If newer packages are installed on an overlay do we generally expect these conda environments to carry the dependencies to support them, or should the dependencies be installed on the overlay itself?

If we allow some "future" dependencies out of necessity it would be good to keep a list of them somewhere. Just a suggestion.

Alternatively, should we pin nslsii to a newer version instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we never went back to https://github.com/conda-forge/nslsii-feedstock/blob/main/recipe/meta.yaml and added the requirement. Please feel free to submit a PR there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seem to be few other dependencies missing in https://github.com/conda-forge/nslsii-feedstock/blob/main/recipe/meta.yaml. For example, httpx, msgpack, msgpack-numpy, packaging, redis-json-dict are also missing. Should they also be added to feedstock? They are all available on conda-forge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's create an issue and address it in 2024-2.4.

- pyfai >=2024.5.0
- pymatgen >=2024.5.1
- pyobjcryst
Expand Down Expand Up @@ -211,7 +216,6 @@ dependencies:
- envisage=6.0.1=pyhd8ed1ab_0
- epics-base=7.0.7.0=h2dfad98_0
- et_xmlfile=1.1.0=pyhd8ed1ab_0
- event-model=1.20.0=pyhd8ed1ab_0
- exceptiongroup=1.2.0=pyhd8ed1ab_2
- executing=2.0.1=pyhd8ed1ab_0
- expat=2.5.0=hcb278e6_1
Expand Down
75 changes: 40 additions & 35 deletions envs/env-py311.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: 2024-2.2-py311-tiled
name: 2024-2.3-py311-tiled
channels:
- conda-forge
dependencies:
Expand Down Expand Up @@ -33,6 +33,7 @@ dependencies:
- cmasher
- conda-pack
- conftrak >=0.0.9
- cookiecutter
- csxtools >=0.2.1
- dash
- dash-bootstrap-components
Expand All @@ -44,14 +45,15 @@ dependencies:
- databroker >=2.0.0b45
- dictdiffer
- diffpy.pdffit2
- diffpy.pdfgui
- diffpy.structure
- discorpy
- distributed
- doi2bib
- dpcmaps
# - edrixs # conflicts caused by the latest numexpr, to resolve later.
- eiger-io
- event-model >=1.20
- event-model >=1.21
- fabio
- ffmpeg >=4.0
- flake8
Expand All @@ -78,6 +80,7 @@ dependencies:
- ispyb
- isstools
- jedi
- julia
- jupyter
- jupyterlab
- ldap3
Expand Down Expand Up @@ -122,6 +125,7 @@ dependencies:
- py-xgboost
- py4xs
- pycentroids
- pycryptodome
- pyepics >=3.4.2
- pyfai >=2024.5.0
- pyfftw
Expand Down Expand Up @@ -199,36 +203,37 @@ dependencies:
# Dependencies from the `nsls2-collection` metapackage #
# #
#***************************************************************************#
- bloptools >=0.7.0
- bluesky-darkframes >=0.6.0
- caproto
- emojis
- happi
- pexpect
# - pydm
- pyolog >=4.5.0
- pyserial
- python-confluent-kafka
- pyzenodo3
- simple-pid
- slack-sdk
# Beamline-specific packages
- hklpy # [linux]
- hxnfly >=0.0.11
- kkcalc
- ppmac
- pychx >=4.3.1
- xpdacq ==1.0.0
# Debugging tools:
- hunter
- logging_tree
# Profiling tools:
- line_profiler
- pyinstrument
- pyperformance
# ML:
- botorch
- gpytorch
- ortools-python
- pytorch
- scikit-optimize
# Those package will be added to the successfully-resolved conda env:
# - bloptools >=0.7.0
# - bluesky-darkframes >=0.6.0
# - caproto
# - emojis
# - happi
# - pexpect
# # - pydm
# - pyolog >=4.5.0
# - pyserial
# - python-confluent-kafka
# - pyzenodo3
# - simple-pid
# - slack-sdk
# # Beamline-specific packages
# - hklpy # [linux]
# - hxnfly >=0.0.11
# - kkcalc
# - ppmac
# - pychx >=4.3.1
# - xpdacq ==1.0.0
# # Debugging tools:
# - hunter
# - logging_tree
# # Profiling tools:
# - line_profiler
# - pyinstrument
# - pyperformance
# # ML:
# - botorch
# - gpytorch
# - ortools-python
# - pytorch
# - scikit-optimize
8 changes: 6 additions & 2 deletions envs/env-py312.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: 2024-2.2-py312-tiled
name: 2024-2.3-py312-tiled
channels:
- conda-forge
dependencies:
Expand Down Expand Up @@ -33,6 +33,7 @@ dependencies:
- cmasher
- conda-pack
- conftrak >=0.0.9
- cookiecutter
- csxtools >=0.2.1
- dash
- dash-bootstrap-components
Expand All @@ -44,14 +45,15 @@ dependencies:
- databroker >=2.0.0b45
- dictdiffer
- diffpy.pdffit2
# - diffpy.pdfgui # no package for py312
- diffpy.structure
- discorpy
- distributed
- doi2bib
- dpcmaps
# - edrixs # conflicts caused by the latest numexpr, to resolve later.
- eiger-io
- event-model >=1.20
- event-model >=1.21
- fabio
- ffmpeg >=4.0
- flake8
Expand All @@ -78,6 +80,7 @@ dependencies:
- ispyb
- isstools
- jedi
- julia
- jupyter
- jupyterlab
- ldap3
Expand Down Expand Up @@ -122,6 +125,7 @@ dependencies:
- py-xgboost
- py4xs
- pycentroids
- pycryptodome
- pyepics >=3.4.2
- pyfai >=2024.5.0
# - pyfftw # no build for py312 as of 2024-06-20
Expand Down