feat: support for python 3.11

.devcontainer.json

@@ -18,7 +18,7 @@
 	// "forwardPorts": [],
 
 	// Use 'postCreateCommand' to run commands after the container is created.
-	"postCreateCommand": "mamba create -y -n haptools -c conda-forge 'poetry>=1.1.15' 'python=3.7' && conda run -n haptools poetry install -E docs -E tests",
+	"postCreateCommand": "mamba env create -n haptools -f dev-env.yml && conda run -n haptools poetry install",
 
 	// Configure tool-specific properties.
 	"customizations": {
@@ -29,7 +29,7 @@
 				"python.defaultInterpreterPath": "/opt/conda/envs/haptools/bin/python",
 				"python.terminal.activateEnvironment": true,
 				"python.terminal.activateEnvInCurrentTerminal": true,
-				"python.venvFolders": ["/opt/conda/envs"],
+				"python.venvFolders": ["/home/vscode/.cache/pypoetry/virtualenvs"],
 				"terminal.integrated.environmentChangesRelaunch": true,
 				"terminal.integrated.hideOnStartup": "always"
 			}

.github/pull_request_template.md

@@ -1,49 +1,13 @@
-Title your PR according to [the conventional commits specification](https://www.conventionalcommits.org/).
-
-If your pull request resolves any open issues, please list them here at the top of the PR like this:
-
-resolves #0
-
-## Overview
-
-Include a one sentence summary of the changes to our code and a separate paragraph explaining relevant motivation and context.
-
-Also, list any dependencies that are required for this change.
-
-## Usage and Documentation
-
-You can call the new command like this...
-Documentation for all of this is linked here...
-
-## Implementation Details
-
-The new command works by calling the following new classes...
-
-### WickedAwesomeClass
-The primary method for this class is...
-
-### HelperForWickedClass
-This works by...
-
-## Tests
-
-Please describe the tests that you ran to verify your changes. All tests should be added to our test suite in the `tests/` directory. Please also list any relevant details for your test configuration (ex: the version of pip you used, if it's relevant).
-
-1. `test_a()`
-    Handles the most basic case where...
-2. `test_b()`
-	Tests that we handle...
-
-## Future work
-I didn't have a chance to do...
-In the future, we may want to refactor for...
-
-
 ## Checklist
 
+* [ ] I've checked to ensure there aren't already other open [pull requests](../../../pulls) for the same update/change
+* [ ] I've prefixed the title of my PR according to [the conventional commits specification](https://www.conventionalcommits.org/). If your PR fixes a bug, please prefix the PR with `fix: `. Otherwise, if it introduces a new feature, please prefix it with `feat: `. If it introduces a breaking change, please add an exclamation before the colon, like `feat!: `. If the scope of the PR changes because of a revision to it, please update the PR title, since the title will be used in our CHANGELOG.
+* [ ] At the top of the PR, I've [listed any open issues that this PR will resolve](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword). For example, "resolves #0" if this PR resolves issue #0
+- [ ] I've explained my changes in a manner that will make it possible for both users and maintainers of haptools to understand them
 * [ ] I have followed the [contributing guidelines](https://haptools.readthedocs.io/en/stable/project_info/contributing.html#how-to-fix-a-bug-or-implement-a-new-feature)
 * [ ] I have adhered to the [style guidelines](https://haptools.readthedocs.io/en/stable/project_info/contributing.html#style)
-* [ ] I have checked to ensure there aren't other open [pull requests](../../../pulls) for the same update/change
-* [ ] (if applicable) I have added changes to the dependencies to the pyproject.toml file, and I've run `poetry lock` to ensure the lock file stays up to date
-
-<!-- For an example that follows this PR outline, see [PR #43](https://github.com/CAST-genomics/haptools/pull/43) -->
+* [ ] I've added tests for any new functionality. Or, if this PR fixes a bug, I've added test(s) that replicate it
+* [ ] I've updated the relevant documentation and checked that the newly built documentation is formatted properly
+* [ ] All functions, modules, classes etc. still conform to [numpy docstring standards](https://numpydoc.readthedocs.io/en/latest/format.html)
+* [ ] (if applicable) I've updated the pyproject.toml file with any changes I've made to haptools's dependencies, and I've run `poetry lock --no-update` to ensure the lock file stays up to date and that our dependencies are locked to their minimum versions
+* [ ] In the body of this PR, I've included a short address to the reviewer highlighting one or two items that might deserve their focus

.github/workflows/tests.yml

@@ -10,13 +10,15 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - { python: "3.7", os: "ubuntu-latest", session: "lint" }
+          - { python: "3.8", os: "ubuntu-latest", session: "lint" }
           - { python: "3.7", os: "ubuntu-latest", session: "tests" }
           - { python: "3.8", os: "ubuntu-latest", session: "tests" }
           - { python: "3.9", os: "ubuntu-latest", session: "tests" }
           - { python: "3.10", os: "ubuntu-latest", session: "tests" }
-          # - { python: "3.10", os: "windows-latest", session: "tests" }
-          - { python: "3.10", os: "macos-latest", session: "tests" }
+          - { python: "3.11", os: "ubuntu-latest", session: "tests" }
+          # - { python: "3.11", os: "windows-latest", session: "tests" }
+          - { python: "3.9", os: "macos-latest", session: "tests" }
+          - { python: "3.8", os: "ubuntu-latest", session: "size" }
 
     env:
       NOXSESSION: ${{ matrix.session }}
@@ -25,90 +27,75 @@ jobs:
 
     steps:
       - name: Check out the repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v3
+      - name: Setup Mambaforge
+        uses: conda-incubator/setup-miniconda@v3
         with:
-          python-version: ${{ matrix.python }}
+          activate-environment: haptools
+          miniforge-variant: Mambaforge
+          auto-activate-base: false
+          miniforge-version: latest
+          use-mamba: true
 
-      - name: Upgrade pip
-        run: |
-          pip install --constraint=.github/workflows/constraints.txt pip
-          pip --version
-      - name: Upgrade pip in virtual environments
-        shell: python
-        run: |
-          import os
-          import pip
-          with open(os.environ["GITHUB_ENV"], mode="a") as io:
-              print(f"VIRTUALENV_PIP={pip.__version__}", file=io)
-      - name: Install Poetry
+      - name: Get Date
+        id: get-date
+        run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
+        shell: bash
+
+      - name: Cache Conda env
+        uses: actions/cache@v3
+        with:
+          path: ${{ env.CONDA }}/envs
+          key:
+            conda-${{ runner.os }}--${{ runner.arch }}--${{ steps.get-date.outputs.today }}-${{ hashFiles('dev-env.yml') }}-${{ env.CACHE_NUMBER }}
+        env:
+          # Increase this value to reset cache if dev-env.yml has not changed
+          CACHE_NUMBER: 0
+        id: cache
+
+      - name: Install dev environment
+        run:
+          mamba env update -n haptools -f dev-env.yml
+        if: steps.cache.outputs.cache-hit != 'true'
+
+      - name: Try to build haptools
+        shell: bash -el {0}
         run: |
-          pipx install --python python3.10 --pip-args=--constraint=.github/workflows/constraints.txt poetry
-          poetry --version
-      - name: Install Nox
+          poetry build --no-ansi
+
+      - name: Check distribution size
+        if: matrix.session == 'size'
         run: |
-          pipx install --python python3.10 --pip-args=--constraint=.github/workflows/constraints.txt nox
-          pipx inject --pip-args=--constraint=.github/workflows/constraints.txt nox nox-poetry
-          nox --version
-      - name: Run Nox
+          du -csh dist/*
+          # check that the generated dist/ directory does not exceed 0.3 MB
+          # if this check fails, it's because you forgot to list large files in a "tool.poetry.exclude" section of our pyproject.toml
+          # https://python-poetry.org/docs/pyproject/#include-and-exclude
+          [ $(du -b dist | cut -f1) -lt 300000 ]
+
+      - name: Run tests with nox
+        if: matrix.session != 'size'
+        shell: bash -el {0}
         run: |
-          nox --python=${{ matrix.python }}
+          nox --verbose --python=${{ matrix.python }}
+
       - name: Upload coverage data
         if: always() && matrix.session == 'tests'
         uses: "actions/upload-artifact@v3"
         with:
           name: coverage-data
           path: ".coverage.*"
+
   large-files:
     name: File sizes
     runs-on: ubuntu-latest
     steps:
       - name: Check out the repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
+
       - name: Check for large files
         uses: actionsdesk/[email protected]
         with:
           token: ${{ secrets.GITHUB_TOKEN }} # Optional
           filesizelimit: 500000b
           labelName: large-files
-
-  # coverage:
-  #   runs-on: ubuntu-latest
-  #   needs: tests
-  #   steps:
-  #     - name: Check out the repository
-  #       uses: actions/checkout@v3
-
-  #     - name: Set up Python
-  #       uses: actions/setup-python@v3
-  #       with:
-  #         python-version: "3.10"
-
-  #     - name: Upgrade pip
-  #       run: |
-  #         pip install --constraint=.github/workflows/constraints.txt pip
-  #         pip --version
-  #     - name: Install Poetry
-  #       run: |
-  #         pipx install --pip-args=--constraint=.github/workflows/constraints.txt poetry
-  #         poetry --version
-  #     - name: Install Nox
-  #       run: |
-  #         pipx install --pip-args=--constraint=.github/workflows/constraints.txt nox
-  #         pipx inject --pip-args=--constraint=.github/workflows/constraints.txt nox nox-poetry
-  #         nox --version
-  #     - name: Download coverage data
-  #       uses: actions/download-artifact@v3
-  #       with:
-  #         name: coverage-data
-
-  #     - name: Combine coverage data and display human readable report
-  #       run: |
-  #         nox --session=coverage
-  #     - name: Create coverage report
-  #       run: |
-  #         nox --session=coverage -- xml
-  #     - name: Upload coverage report
-  #       uses: codecov/[email protected]

.gitignore

@@ -8,7 +8,7 @@ docs/_build/
 __pycache__
 # pytest and testing-related caches
 .pytest_cache
-.coverage
+.coverage*
 .nox
 # poetry
 dist/

.readthedocs.yaml

@@ -2,23 +2,26 @@
 # Read the Docs configuration file
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
 
-# Note: I used https://github.com/readthedocs/readthedocs.org/issues/4912#issuecomment-664002569 for inspiration
+# Note: I used https://docs.readthedocs.io/en/stable/build-customization.html#install-dependencies-with-poetry for inspiration
 
 version: 2
 
 build:
   os: "ubuntu-22.04"
   tools:
-    python: "3.7"
+    python: "3.8"
+  jobs:
+    post_create_environment:
+      # Install poetry
+      # https://python-poetry.org/docs/#installing-manually
+      - pip install poetry
+      # Tell poetry to not use a virtual environment
+      - poetry config virtualenvs.create false
+    post_install:
+      # Install dependencies with 'docs' dependency group
+      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
+      - poetry install --only main,docs
 
 sphinx:
   configuration: docs/conf.py
   fail_on_warning: true
-
-python:
-  install:
-    - method: pip
-      path: .
-      extra_requirements:
-        - docs
-        - files

dev-env.yml

@@ -0,0 +1,13 @@
+channels:
+  - conda-forge
+  - bioconda
+  - nodefaults
+dependencies:
+  - conda-forge::python=3.8 # the lowest version of python that we formally support
+  - conda-forge::pip==23.3.2
+  - conda-forge::poetry==1.7.1
+  - conda-forge::nox==2023.04.22
+  - conda-forge::poetry-plugin-export==1.6.0
+  - pip:
+    - nox-poetry==1.0.3
+    - poetry-conda==0.1.1

docs/project_info/contributing.rst

@@ -70,23 +70,19 @@ Follow these steps to set up a development environment.
 
     .. code-block:: bash
 
-        conda create -n haptools-dev -c conda-forge 'poetry>=1.1.15' 'python=3.7'
-2. Activate the environment
+        conda env create -n haptools -f dev-env.yml
+2. Install haptools and its dependencies into a separate environment managed by ``poetry``
 
     .. code-block:: bash
 
-        conda activate haptools-dev
-3. Install our dependencies
+        conda run -n haptools poetry install
 
-    .. code-block:: bash
-
-        poetry install -E docs -E tests
+3. Now, whenever you'd like to run/import ``haptools`` or ``pytest``, you will first need to activate both environments
 
-Now, try importing ``haptools`` or running it on the command line.
-
-.. note::
-   If you run into an attribute error ``module 'distutils' has no attribute 'util'`` see `this workaround <https://github.com/python-poetry/poetry/issues/3336#issuecomment-831789763>`_.
+    .. code-block:: bash
 
+        conda activate haptools
+        poetry shell
 
 ---------------------
 Managing Dependencies

haptools/clump.py

@@ -426,7 +426,8 @@ def _FilterGts(candidate_gt, index_gt, log):
     once converted to uint8 and sum alleles together.
     """
     # Check Alleles for invalid values and remove samples from both sets of alleles
-    valid_gts = np.all(candidate_gt < 254, axis=1) & np.all(index_gt < 254, axis=1)
+    miss = np.iinfo(np.uint8).max - 1
+    valid_gts = np.all(candidate_gt < miss, axis=1) & np.all(index_gt < miss, axis=1)
     candidate_gt = candidate_gt[valid_gts, :]
     index_gt = index_gt[valid_gts, :]
     log.debug(f"Valid Genotype Indices: {valid_gts}")