Skip to content

Commit

Permalink
remove boa support! (#926)
Browse files Browse the repository at this point in the history 
* moved to rattler-build
  • Loading branch information
@DerThorsten
DerThorsten authored May 10, 2024
1 parent 7549205 commit 5d7c065
Show file tree
Hide file tree
Showing 173 changed files with 2,853 additions and 4,286 deletions.
28 changes: 2 additions & 26 deletions .github/workflows/build_recipes.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,19 +71,6 @@ jobs:
run: |
cp $GITHUB_WORKSPACE/.github/workflows/.condarc $HOME
cp $GITHUB_WORKSPACE/conda_build_config.yaml $HOME
playwright install
################################################################
# Install / clone custom non-master things
# for pip, we use --no-deps --ignore-installed
# and make sure that all dependencies are already contained
# in the conda env named "ci-env"
################################################################
- name: Install custom boa
shell: bash -el {0}
run: |
python -m pip install git+https://github.com/DerThorsten/boa.git@python_api_v2 --no-deps --ignore-installed
################################################################
# PUSH
Expand All @@ -92,7 +79,7 @@ jobs:
shell: bash -el {0}
if: github.event_name == 'push'
run: |
python -m emci build changed $GITHUB_WORKSPACE/ origin/main~1 origin/main
python -m emci build changed $GITHUB_WORKSPACE/ origin/main~1 origin/main --n-retry=3
################################################################
# PULL_REQUEST
Expand All @@ -101,7 +88,7 @@ jobs:
shell: bash -el {0}
if: github.event_name == 'pull_request'
run: |
python -m emci build changed $GITHUB_WORKSPACE/ origin/main HEAD
python -m emci build changed $GITHUB_WORKSPACE/ origin/main HEAD --n-retry=3
################################################################
# UPLOAD
Expand All @@ -110,17 +97,6 @@ jobs:
if: (github.event_name == 'push' && github.repository == 'emscripten-forge/recipes')
shell: bash -el {0}
run: |
# loop over {emscripten-wasm32, linux-64, noarch}
for platform in emscripten-wasm32 linux-64 noarch; do
# loop over all packages
mkdir -p ${CONDA_PREFIX}/conda-bld/${platform}
for package in $(ls ${CONDA_PREFIX}/conda-bld/${platform}/*.tar.bz2); do
echo "Uploading ${package} for ${platform} (build with boa)"
QUETZ_API_KEY=${{ secrets.QUETZ_API_KEY}} quetz-client -u https://beta.mamba.pm post_file_to_channel emscripten-forge ${package}
done
done
# loop over {emscripten-wasm32, linux-64, noarch}
for platform in emscripten-wasm32 linux-64 noarch; do
mkdir -p ${GITHUB_WORKSPACE}/output/${platform}
Expand Down
50 changes: 5 additions & 45 deletions .github/workflows/build_simple.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ jobs:
run: |
cp $GITHUB_WORKSPACE/.github/workflows/.condarc $HOME
cp $GITHUB_WORKSPACE/conda_build_config.yaml $HOME
playwright install
################################################################
# EMSDK
Expand All @@ -65,73 +64,34 @@ jobs:
run: |
./emsdk/setup_emsdk.sh ${{ matrix.emsdk_ver }} $(pwd)/emsdk_install
################################################################
# Install / clone custom non-master things
# for pip, we use --no-deps --ignore-installed
# and make sure that all dependencies are already contained
# in the conda env named "ci-env"
################################################################
- name: Install custom boa
shell: bash -el {0}
run: |
python -m pip install git+https://github.com/DerThorsten/boa.git@python_api_v2 --no-deps --ignore-installed
################################################################
# Build compiler pkg
################################################################
- name: Build emscripten compiler
shell: bash -el {0}
run: |
FORCE_RATTLER=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes/emscripten_emscripten-wasm32
python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes/emscripten_emscripten-wasm32 --n-retry=3
################################################################
# Build compiler pkg with legacy boa
################################################################
- name: Build emscripten compiler
shell: bash -el {0}
run: |
FORCE_BOA=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes/emscripten_emscripten-wasm32
################################################################
# Build pytester
################################################################
- name: Build pytester
shell: bash -el {0}
run: |
FORCE_RATTLER=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes/pytester
python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes/pytester
################################################################
# Build a simple package
# Build a regex package
################################################################
- name: Build regex
shell: bash -el {0}
run: |
FORCE_RATTLER=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes_emscripten/regex --emscripten-wasm32
python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes_emscripten/regex --emscripten-wasm32 --n-retry=3

################################################################
# Build a package which is **not** on emscripten forge
################################################################
- name: Build package not on emscripten-forge (advanced)
shell: bash -el {0}
run: |
FORCE_RATTLER=1 python -m emci build explicit $GITHUB_WORKSPACE/test_recipes/only_py_tests --emscripten-wasm32
################################################################
# Build a simple package
################################################################
- name: Build regex
shell: bash -el {0}
run: |
FORCE_RATTLER=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes_emscripten/highs --emscripten-wasm32
################################################################
# Build a simple package with legacy boa
################################################################
- name: Build regex with legacy boa
shell: bash -el {0}
run: |
FORCE_BOA=1 python -m emci build explicit $GITHUB_WORKSPACE/recipes/recipes_emscripten/regex --emscripten-wasm32
python -m emci build explicit $GITHUB_WORKSPACE/test_recipes/only_py_tests --emscripten-wasm32 --n-retry=3
2 changes: 1 addition & 1 deletion .github/workflows/run_with_pixi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ jobs:
pixi-version: v0.20.1
- run: |
pixi run setup
pixi run build-emscripten-wasm32-pkg recipes/recipes_emscripten/regex
pixi run build-emscripten-wasm32-pkg recipes/recipes_emscripten/regex
5 changes: 4 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,7 @@ Untitled.ipynb
# pixi folder
.pixi/
# rattler-build output folder
output/
output/

# emscripten forge install
emscripten_forge_emsdk_install/
103 changes: 15 additions & 88 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@

Build wasm/emscripten packages with conda/mamba/boa.
This repository consists of recipes for conda packages for emscripten.
Most of the recipes have been ported from [pyodide](https://pyodide.org/en/stable/).

While we already have a lot of packages built, this is still a big work in progress.

> **Note**
> The recipes used in this repository follow the [Boa recipe specification](https://boa-build.readthedocs.io/en/latest/recipe_spec.html).
> We removed boa support from this repository. We are now using [rattler-build](https://github.com/prefix-dev/rattler-build)
> **Note**
> The recipes used in this repository follow the [rattler-build recipe format](https://github.com/prefix-dev/rattler-build?tab=readme-ov-file#the-recipe-format).
## Installing Packages
We recommend using micromamba to install packages from this channel.
Expand All @@ -27,23 +28,18 @@ micromamba create -n my-channel-name \
To add a new package to emscripten-forge, just create a Pull Request in this repository.
This PR should add a new folder in the [recipe_emscripten](https://github.com/emscripten-forge/recipes/tree/main/recipes/recipes_emscripten) folder
containing the recipe of your package.
Good example recipes are:
A good example recipes is:

* [cffi_example](https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/cffi_example/recipe.yaml) for a `cffi` based package.
* [xeus-python](https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/xeus-python/recipe.yaml) for a package with a CMake-based build system
* [regex](https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/regex/recipe.yaml) .

Once the PR is merged, the package is built and uploaded to https://beta.mamba.pm/channels/emscripten-forge


## Local Builds
We are currently phasing out the use of `boa` in favor of `rattler-build`. Yet we still support `boa` builds
as not all recipes have been ported to `rattler-build` yet.
For each recipe there can be a `rattler_recipe.yaml` and a `recipe.yaml` file. The `rattler_recipe.yaml` file is used for rattler-build and the `recipe.yaml` file is used for boa builds.
Once all recipes have been ported to `rattler-build`, we will remove the `recipe.yaml` files and rename the `rattler_recipe.yaml` files to `recipe.yaml`.

### Local Builds with pixi rattler-build

To build a package locally, the easiest way is to use `pixi`
## Local Builds

To build a package locally, the easiest way is to use `pixi` (see [/pixi.sh/latest/#installation](https://pixi.sh/latest/#installation) for installation instructions).

```bash
# this only needs to be done once
Expand All @@ -53,7 +49,7 @@ pixi run build-emscripten-wasm32-pkg recipes/recipes_emscripten/regex
```


### Local Builds with rattler-build
## Local Builds with rattler-build
Only recipes with a `rattler_recipe.yaml` file can be built with `rattler-build`.


Expand All @@ -76,90 +72,21 @@ Ie if you open a new terminal, you have to activate the environment again with `
./emsdk/setup_emsdk.sh 3.1.45 ~/emsdk
```

**2b** Build compiler packages / meta packages with for MacOS:
**2b** Build compiler packages / meta packages :

This is only needed for MacOS. On Linux, the compiler packages are already built and available in the `emscripten-forge` channel.
```bash
rattler-build build --recipe recipes/recipes/emscripten_emscripten-wasm32/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m rattler_conda_build_config.yaml
rattler-build build --recipe recipes/recipes/cross-python_emscripten-wasm32/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m rattler_conda_build_config.yaml
rattler-build build --recipe recipes/recipes/pytester/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m rattler_conda_build_config.yaml
rattler-build build --recipe recipes/recipes/emscripten_emscripten-wasm32/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m conda_build_config.yaml
rattler-build build --recipe recipes/recipes/cross-python_emscripten-wasm32/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m conda_build_config.yaml
rattler-build build --recipe recipes/recipes/pytester/rattler_recipe.yaml -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m conda_build_config.yaml
```

**3** Build packages with `rattler-build`:

```bash
rattler-build build --recipe recipes/recipes_emscripten/regex/rattler_recipe.yaml --target-platform=emscripten-wasm32 -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m rattler_conda_build_config.yaml
```


### Local Builds with boa (this is deprecated)
Local builds are useful for testing new recipes or debugging build issues, but the setup is a bit more involved and will only work for Linux and MacOS. Local builds on Windows are not yet supported.

**1** Create a new conda environment from `ci_env.yml` and install playwright in this environment:
On a Linux system this can be done with:
```bash
micromamba create -n emscripten-forge -f ci_env.yml --yes
micromamba activate emscripten-forge
playwright install
```
On a MacOS system, this can be done with:
```bash
micromamba create -n emscripten-forge -f ci_env_macos.yaml --yes
micromamba activate emscripten-forge
python pip install playwright
playwright install
rattler-build build --recipe recipes/recipes_emscripten/regex/rattler_recipe.yaml --target-platform=emscripten-wasm32 -c https://repo.mamba.pm/emscripten-forge -c conda-forge -c microsoft -m conda_build_config.yaml
```

**All further steps should be executed in this environment.**
Ie if you open a new terminal, you have to activate the environment again with `micromamba activate emscripten-forge`.

**2** Setup emsdk:
We currently need a patched version of emsdk. This is because emscripten had some regressions in the `3.1.45` release wrt. dynamic loading of shared libraries. We use the `./emsdk/setup_emsdk.sh` which takes
two arguments: the emsdk version and the path where emsdk should be installed.
In this example we choose `~/emsdk` as the installation path. You have to use version `3.1.45`.
```bash
./emsdk/setup_emsdk.sh 3.1.45 ~/emsdk
```

**3** Install a custom version of `boa`
```bash
python -m pip install git+https://github.com/DerThorsten/boa.git@python_api_v2 --no-deps --ignore-installed
```

**4** Add emscripten-forge channel to `.condarc`.
Create a `.condarc` file in your home directory with the following content:
```bash
channels:
- "https://repo.mamba.pm/emscripten-forge"
- conda-forge
```

**5a** Build a package (simple version):
To build a package, run this from the root of the repository (replace `regex` with the package you want to build)
```bash
boa build --target-platform=emscripten-wasm32 recipes/recipes_emscripten/regex -m conda_build_config.yaml
```
This should work in principle, but will not run the python tests of the package.

**5b** Build a python package and run tests:
For web assembly packages, we need to run the tests in a browser. This is done via playwright.
We have a custom hacky script which builds the packages via boa and then runs the playwright tests.
To build a package and run the tests, run this from the root of the repository (replace `regex` with the package you want to build)

```bash
python -m emci build explicit recipes/recipes_emscripten/regex --emscripten-wasm32
```

**6** Building multiple local packages which depend on each other:

If you want to build multiple packages which depend on each other, you have to add the `local-channel` to your `.condarc` file. Since we are building from the `emscripten-forge` environment, we need to modify the `.condarc` file in the `emscripten-forge` environment. You have to add the the `conda-bld` dir
of the `emscripten-forge` environment to the `.condarc` file. This is usually located in `~/micromamba/envs/emscripten-forge/conda-bld` on unix systems.
```bash
channels:
- /home/your-user-name/micromamba/envs/emscripten-forge/conda-bld
- "https://repo.mamba.pm/emscripten-forge"
- conda-forge
```

## Outlook

Expand Down
40 changes: 3 additions & 37 deletions ci_env.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,42 +3,8 @@ channels:
- conda-forge
dependencies:
- python
- pip
- empack >=3,<4
- colorama
- ruamel
- mamba >= 0.23.1
- micromamba >= 0.23.1
- cython
- patchelf
- liblief
- py-lief
- conda >= 23.7.4
- conda-build >= 3.26.1, <3.28
- pyyaml
- click == 8.0.4
- typer >= 0.7.0
- quetz-client >= 0.5.0
- nodejs >= 18.7.0
- pydantic
- pluggy ==1.0.0
- pytest
- networkx
- pyjs_code_runner >= 2,<3
- cxx-compiler
- c-compiler
- cmake
- make
- wget
# hack to get in boas dependencies
- ruamel.yaml < 0.18
- jinja2
- rich
- prompt_toolkit
- json5
- jsonschema
- joblib
- watchgod
- dataclasses
- typer
- retry
- rattler-build >= 0.14.0
- microsoft::playwright < 1.42
- pixi
Loading

0 comments on commit 5d7c065

Please sign in to comment.