update plugin to include docs and fix docker build

It is really hard to install python 3.12 alongside the latest ubuntu (jammy) and flux for it. Signed-off-by: vsoch <[email protected]>
snakemake · Jan 11, 2024 · 83bae3f · 83bae3f
1 parent 8b38181
commit 83bae3f
Show file tree

Hide file tree

Showing 7 changed files with 342 additions and 84 deletions.
diff --git a/README.md b/README.md
@@ -101,5 +101,11 @@ plugin for use or development (that doesn't need to be added to upstream snakema
 
 ## Developer
 
+To do the same run but bind the local plugin directory:
+
+```bash
+docker run -it -v $PWD/:/home/fluxuser/plugin flux-snake bash
+```
+
 The instructions for creating and scaffolding this plugin are [here](https://github.com/snakemake/poetry-snakemake-plugin#scaffolding-an-executor-plugin).
-Instructions for writing your plugin with examples are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).
+Instructions for writing your plugin with examples are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).
diff --git a/docs/further.md b/docs/further.md
@@ -0,0 +1,111 @@
+# Snakemake Executor Flux
+
+This is an example implementation for an external snakemake plugin.
+Since we already have one for Flux (and it can run in a container) the example
+is for Flux. You can use this repository as a basis to design your own executor to work
+with snakemake!
+
+## Usage
+
+### Tutorial
+
+For this tutorial you will need Docker installer.
+
+[Flux-framework](https://flux-framework.org/) is a flexible resource scheduler that can work on both high performance computing systems and cloud (e.g., Kubernetes).
+Since it is more modern (e.g., has an official Python API) we define it under a cloud resource. For this example, we will show you how to set up a "single node" local
+Flux container to interact with snakemake using the plugin here. You can use the [Dockerfile](examples/Dockerfile) that will provide a container with Flux and snakemake
+Note that we install from source and bind to `/home/fluxuser/snakemake` with the intention of being able to develop (if desired).
+
+First, build the container:
+
+```bash
+$ docker build -f example/Dockerfile -t flux-snake .
+```
+
+We will add the plugin here to `/home/fluxuser/plugin`, install it, and shell in as the fluxuser to optimally interact with flux.
+After the container builds, shell in:
+
+```bash
+$ docker run -it flux-snake bash
+```
+
+And start a flux instance:
+
+```bash
+$ flux start --test-size=4
+```
+
+Go into the examples directory (where the Snakefile is) and run snakemake, targeting your executor plugin.
+
+```bash
+$ cd ./example
+
+# This says "use the custom executor module named snakemake_executor_plugin_flux"
+$ snakemake --jobs 1 --executor flux
+```
+```console
+Building DAG of jobs...
+Using shell: /bin/bash
+Job stats:
+job                         count    min threads    max threads
+------------------------  -------  -------------  -------------
+all                             1              1              1
+multilingual_hello_world        2              1              1
+total                           3              1              1
+
+Select jobs to execute...
+
+[Fri Jun 16 19:24:22 2023]
+rule multilingual_hello_world:
+    output: hola/world.txt
+    jobid: 2
+    reason: Missing output files: hola/world.txt
+    wildcards: greeting=hola
+    resources: tmpdir=/tmp
+
+Job 2 has been submitted with flux jobid ƒcjn4t3R (log: .snakemake/flux_logs/multilingual_hello_world/greeting_hola.log).
+[Fri Jun 16 19:24:32 2023]
+Finished job 2.
+1 of 3 steps (33%) done
+Select jobs to execute...
+
+[Fri Jun 16 19:24:32 2023]
+rule multilingual_hello_world:
+    output: hello/world.txt
+    jobid: 1
+    reason: Missing output files: hello/world.txt
+    wildcards: greeting=hello
+    resources: tmpdir=/tmp
+
+Job 1 has been submitted with flux jobid ƒhAPLa79 (log: .snakemake/flux_logs/multilingual_hello_world/greeting_hello.log).
+[Fri Jun 16 19:24:42 2023]
+Finished job 1.
+2 of 3 steps (67%) done
+Select jobs to execute...
+
+[Fri Jun 16 19:24:42 2023]
+localrule all:
+    input: hello/world.txt, hola/world.txt
+    jobid: 0
+    reason: Input files updated by another job: hello/world.txt, hola/world.txt
+    resources: tmpdir=/tmp
+
+[Fri Jun 16 19:24:42 2023]
+Finished job 0.
+3 of 3 steps (100%) done
+Complete log: .snakemake/log/2023-06-16T192422.186675.snakemake.log
+```
+
+And that's it! Continue reading to learn more about plugin design, and how you can also design your own executor
+plugin for use or development (that doesn't need to be added to upstream snakemake).
+
+## Developer
+
+To do the same run but bind the local plugin directory:
+
+```bash
+docker run -it -v $PWD/:/home/fluxuser/plugin flux-snake bash
+```
+
+The instructions for creating and scaffolding this plugin are [here](https://github.com/snakemake/poetry-snakemake-plugin#scaffolding-an-executor-plugin).
+Instructions for writing your plugin with examples are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).
diff --git a/docs/intro.md b/docs/intro.md
@@ -0,0 +1 @@
+This is the [Flux Framework](https://flux-framework.org) external executor plugin for snakemake.
diff --git a/example/Dockerfile b/example/Dockerfile
@@ -1,29 +1,209 @@
-FROM fluxrm/flux-sched:jammy
-# This is run from the repository root
-# docker build -t flux-snake .
-# docker run -it flux-snake
-USER root
-ENV PATH=/opt/conda/bin:/root/.local/bin:$PATH
-ENV FLUX_MODULE_PATH=${FLUX_MODULE_PATH}:/usr/lib/x86_64-linux-gnu/flux/modules
+FROM ubuntu:jammy
 
-# Note this is a custom branch that currently has the functionality for plugins
-RUN apt-get update && apt-get install -y python3-pip python3-venv
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git \
+        libfftw3-dev libfftw3-bin pdsh libfabric-dev libfabric1 \
+        dnsutils telnet strace cmake git g++ openmpi-bin \
+        openmpi-common wget curl unzip libopenmpi-dev \
+        software-properties-common gpg-agent build-essential
+
+RUN add-apt-repository ppa:deadsnakes/ppa && apt-get update && \
+    apt install -y python3.12 python3.12-dev
+
+RUN update-alternatives --install /usr/bin/python3 python /usr/bin/python3.12 1 && \
+    update-alternatives --install /usr/bin/python python3 /usr/bin/python3.12 1
+
+RUN wget https://bootstrap.pypa.io/get-pip.py && \
+    python3 get-pip.py
+
+RUN pip3 install -U git+https://github.com/snakemake/snakemake-interface-common@main && \
+    pip3 install -U git+https://github.com/snakemake/snakemake-interface-executor-plugins && \
+    pip3 install -U git+https://github.com/snakemake/snakemake-interface-storage-plugins@main && \
+    pip3 install -U git+https://github.com/snakemake/snakemake-storage-plugin-s3@main && \
+    pip3 install -U git+https://github.com/snakemake/snakemake-storage-plugin-gcs@main && \
+    pip3 install -U git+https://github.com/snakemake/snakemake@main
+
+# This first section from src/test/docker/bionic/Dockerfile in flux-core
+# https://github.com/flux-framework/flux-core/blob/master/src/test/docker/bionic/Dockerfile
+RUN apt-get update && \
+    apt-get -qq install -y --no-install-recommends \
+        apt-utils && \
+    rm -rf /var/lib/apt/lists/*
+
+# Utilities
+RUN apt-get update && \
+    apt-get -qq install -y --no-install-recommends \
+        locales \
+        ca-certificates \
+        curl \
+        wget \
+        man \
+        git \
+        flex \
+        ssh \
+        sudo \
+        vim \
+        luarocks \
+        munge \
+        lcov \
+        ccache \
+        lua5.2 \
+        mpich \
+        valgrind \
+        jq && \
+    rm -rf /var/lib/apt/lists/*
+
+# Compilers, autotools
+RUN apt-get update && \
+    apt-get -qq install -y --no-install-recommends \
+        build-essential \
+        pkg-config \
+        autotools-dev \
+        libtool \
+	libffi-dev \
+        autoconf \
+        automake \
+        make \
+        cmake \
+        clang \
+        clang-tidy \
+        gcc \
+        g++ && \
+    rm -rf /var/lib/apt/lists/*
+
+# Other deps
+RUN apt-get update && \
+    apt-get -qq install -y --no-install-recommends \
+        libsodium-dev \
+        libzmq3-dev \
+        libczmq-dev \
+        libjansson-dev \
+        libmunge-dev \
+        libncursesw5-dev \
+        liblua5.2-dev \
+        liblz4-dev \
+        libsqlite3-dev \
+        uuid-dev \
+        libhwloc-dev \
+        libmpich-dev \
+        libs3-dev \
+        libevent-dev \
+        libarchive-dev \
+        libpam-dev && \
+    rm -rf /var/lib/apt/lists/*
+
+# Testing utils and libs
+RUN apt-get update && \
+    apt-get -qq install -y --no-install-recommends \
+        faketime \
+        libfaketime \
+        pylint \
+        cppcheck \
+        enchant-2 \
+        aspell \
+        aspell-en && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN locale-gen en_US.UTF-8
+
+# NOTE: luaposix installed by rocks due to Ubuntu bug: #1752082 https://bugs.launchpad.net/ubuntu/+source/lua-posix/+bug/1752082
+RUN luarocks install luaposix
+
+
+# Install openpmix, prrte
+WORKDIR /opt/prrte
+RUN git clone https://github.com/openpmix/openpmix.git && \
+    git clone https://github.com/openpmix/prrte.git && \
+    ls -l && \
+    set -x && \
+    cd openpmix && \
+    git checkout fefaed568f33bf86f28afb6e45237f1ec5e4de93 && \
+    ./autogen.pl && \
+    PYTHON=/usr/bin/python3 ./configure --prefix=/usr --disable-static && make -j 4 install && \
+    ldconfig && \
+    cd .. && \
+    cd prrte && \
+    git checkout 477894f4720d822b15cab56eee7665107832921c && \
+    ./autogen.pl && \
+    PYTHON=/usr/bin/python3 ./configure --prefix=/usr && make -j 4 install && \
+    cd ../.. && \
+    rm -rf prrte
+
+ENV LANG=C.UTF-8
+
+# This is from the docker check script (run interactively during a test)
+# https://github.com/flux-framework/flux-core/blob/master/src/test/docker/checks/Dockerfile
+ARG USER=fluxuser
+ARG UID=1000
+ARG GID=1000
+
+# Install flux-security by hand for now:
+#
+WORKDIR /opt
+RUN git clone --depth 1 https://github.com/flux-framework/flux-security && \
+    cd flux-security && \
+    ./autogen.sh && \
+    PYTHON=/usr/bin/python3 ./configure --prefix=/usr --sysconfdir=/etc && \
+    make && \
+    make install && \
+    cd .. && \
+    rm -rf flux-security
+
+# Add configured user to image with sudo access:
+#
+RUN set -x && groupadd -g $UID $USER && \
+    useradd -g $USER -u $UID -d /home/$USER -m $USER && \
+    printf "$USER ALL= NOPASSWD: ALL\\n" >> /etc/sudoers
+
+# Setup MUNGE directories & key
+RUN mkdir -p /var/run/munge && \
+    dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key && \
+    chown -R munge /etc/munge/munge.key /var/run/munge && \
+    chmod 600 /etc/munge/munge.key
+
+# Build flux core
+RUN python3 -m pip install cffi ply
+RUN git clone https://github.com/flux-framework/flux-core && \
+    cd flux-core && \
+    ./autogen.sh && \
+    PYTHON=/usr/bin/python3 PYTHON_PREFIX=PYTHON_EXEC_PREFIX=/usr/lib/python3.12/site-packages ./configure \
+        --prefix=/usr \
+        --sysconfdir=/etc \
+        --with-systemdsystemunitdir=/etc/systemd/system \
+        --localstatedir=/var \
+        --with-flux-security && \
+    make clean && \
+    make && \
+    sudo make install
+
+# This is from the same src/test/docker/bionic/Dockerfile but in flux-sched
+# Flux-sched deps
+RUN sudo apt-get update
+RUN sudo apt-get -qq install -y --no-install-recommends \
+	libboost-graph-dev \
+	libboost-system-dev \
+	libboost-filesystem-dev \
+	libboost-regex-dev \
+	libyaml-cpp-dev \
+	libedit-dev
+
+ENV LD_LIBRARY_PATH=/opt/conda/lib:$LD_LIBRARY_PATH
+
+# Build Flux Sched	
+# https://github.com/flux-framework/flux-sched/blob/master/src/test/docker/docker-run-checks.sh#L152-L158
+RUN git clone https://github.com/flux-framework/flux-sched && \
+    cd flux-sched && \
+    git fetch && \
+    ./autogen.sh && \
+    PYTHON=/usr/bin/python3 ./configure --prefix=/usr --sysconfdir=/etc \
+       --with-systemdsystemunitdir=/etc/systemd/system \
+       --localstatedir=/var && \
+    make && \
+    sudo make install && \
+    ldconfig
 
 WORKDIR /home/fluxuser/plugin
 COPY . /home/fluxuser/plugin
-RUN chown -R fluxuser /home/fluxuser
-
-ENV PATH=/opt/conda/bin:/home/fluxuser/.local/bin:$PATH
-USER fluxuser
-
-# Run/install poetry as fluxuser
-RUN python3 -m pip install --user pipx && \
-    python3 -m pipx ensurepath && \
-    pipx install poetry && \
-    sudo ln -s /bin/python3 /bin/python
-
-RUN poetry lock && \
-    poetry install && \
-    pip install git+https://github.com/vsoch/snakemake@update-lazy-property-import && \
-    pip install -e . && \
-    sudo chown -R fluxuser /home/fluxuser
+
+RUN pip install .
diff --git a/pyproject.toml b/pyproject.toml
@@ -5,21 +5,27 @@ description = "A snakemake executor plugin for the Flux scheduler"
 authors = ["vsoch <[email protected]>"]
 readme = "README.md"
 packages = [{include = "snakemake_executor_plugin_flux"}]
+license = "MIT"
+repository = "https://github.com/snakemake/snakemake-executor-plugin-flux"
+documentation = "https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/flux.html"
+keywords = ["snakemake", "plugin", "executor", "flux", "flux-framework"]
 
 [tool.poetry.dependencies]
 python = "^3.9"
-snakemake-interface-common = "^1.10.0"
-# snakemake-interface-executor-plugins = "^5.0.2"
-snakemake-interface-executor-plugins = { git = "https://github.com/snakemake/snakemake-interface-executor-plugins", branch = "main" }
+snakemake-interface-common = "^1.14.0"
+snakemake-interface-executor-plugins = "^8.1.1"
+
+[tool.coverage.run]
+omit = [".*", "*/site-packages/*", "Snakefile"]
 
 [tool.poetry.group.dev.dependencies]
 black = "^23.9.1"
 flake8 = "^6.1.0"
 coverage = "^7.3.1"
 pytest = "^7.4.2"
-# snakemake = "^7.32.4"
-snakemake = { git = "https://github.com/vsoch/snakemake", branch = "update-lazy-property-import" }
+snakemake = {git = "https://github.com/snakemake/snakemake.git", branch="main"}
 
 [build-system]
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"
+
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		This is the [Flux Framework](https://flux-framework.org) external executor plugin for snakemake.