[CIVIS-2325] DEP upgrade to Python 3.12, update dependency versions (#94

)

* DEP use python 3.12, update dependency versions

* TST drop legacy test for numpy not using mkl

* MAINT update changelog

* DEP refresh req files for requests

* TST set up test suite

* MAINT add github templates

* TST close file object properly

* TST run test suite in verbose mode

.circleci/config.yml

@@ -1,29 +1,17 @@
-version: 2
+version: 2.1
 jobs:
   build:
     docker:
-      - image: circleci/python:3.7.0
+      - image: cimg/python:3.12
     steps:
       - checkout
       - setup_remote_docker
       - run:
           name: Build container
-          command: docker build -t ds-python .
+          command: docker build --target test -t ds-python .
       - run:
           name: Verify build completed
           command: docker run ds-python /bin/bash -c "echo BUILDS OK"
       - run:
-          name: Check that scipy links to OpenBLAS
-          command: docker run ds-python python -c "from scipy.linalg import _fblas"
-      - run:
-          name: Check that numpy imports from bash shell
-          command: docker run -t ds-python /bin/bash -c "python -c 'import numpy'"
-      - run:
-          name: Verify that numpy does not link to MKL
-          command: docker run ds-python python -c "from numpy.distutils import system_info; assert system_info.get_info('mkl') == {}"
-      - run:
-          name: Validate compatability of civis-python with other packages
-          command: docker run ds-python /bin/bash -c "python -c 'import civis'"
-      - run:
-          name: Run numpy unit tests
-          command: docker run ds-python /bin/bash -c 'pip install pytest hypothesis; python -c "import numpy; numpy.test()"'
+          name: Run tests
+          command: docker run ds-python /test_image.py -vv

.circleci/test_image.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+
+import os
+import re
+import unittest
+
+
+class TestImage(unittest.TestCase):
+
+    def test_version(self):
+        version_in_env_var = os.getenv("VERSION")
+        major = os.getenv("VERSION_MAJOR")
+        minor = os.getenv("VERSION_MINOR")
+        micro = os.getenv("VERSION_MICRO")
+        self.assertTrue(major.isdigit())
+        self.assertTrue(minor.isdigit())
+        self.assertTrue(micro.isdigit())
+        self.assertEqual(version_in_env_var, f"{major}.{minor}.{micro}")
+
+        with open("CHANGELOG.md") as changelog:
+            version_in_changelog = re.search(
+                r"##\s+\[(\d+\.\d+\.\d+)]", changelog.read()
+            ).groups()[0]
+        self.assertEqual(version_in_changelog, version_in_env_var)
+
+    def test_scipy_links_to_openblas(self):
+        from scipy.linalg import _fblas  # noqa: F401
+
+    def test_numpy_can_import(self):
+        import numpy as np  # noqa: F401
+
+    def test_civis_can_import(self):
+        import civis  # noqa: F401
+
+
+if __name__ == "__main__":
+    unittest.main()

.github/ISSUE_TEMPLATE/general.md

@@ -0,0 +1,11 @@
+---
+name: General
+about: Ask a question, report a potential issue, etc.
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Note:** Civis employees should _not_ use the GitHub Issues feature at the public "civis-python" codebase
+to file a ticket, and should instead use the internal ticketing system.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,9 @@
+
+
+---
+
+- [ ] (For Civis employees only) Reference to a relevant ticket in the pull request title
+- [ ] Changelog entry added to `CHANGELOG.md` at the repo's root level
+- [ ] Description of change in the pull request description
+- [ ] If applicable, unit tests have been added and/or updated
+- [ ] The CircleCI builds have all passed

CHANGELOG.md

@@ -9,6 +9,18 @@ Version number changes (major.minor.micro) in this package denote the following:
 
 ## Unreleased
 
+## [7.1.0]
+- Python updated to v3.12.3
+- Core dependencies updated to latest versions:
+  * awscli 1.29.5 -> 1.32.109
+  * boto3 1.28.5 -> 1.34.109
+  * civis 1.16.1 -> 2.0.0
+  * numpy 1.25.1 -> 1.26.4
+  * pandas 2.0.3 -> 2.2.2
+  * requests 2.31.0 -> 2.32.2
+  * scikit-learn 1.3.0 -> 1.5.0
+  * scipy 1.11.1 -> 1.13.0
+
 ## [7.0.0]
 - Python updated to v3.11.4
 - Refactors Dockerfile to use the official Python docker image

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.11.4-bookworm
+FROM python:3.12.3-slim AS production
 
 LABEL maintainer = [email protected]
 
@@ -35,7 +35,14 @@ RUN pip install -r requirements-full.txt && \
 # https://github.com/joblib/joblib/blob/0.11/joblib/parallel.py#L328L342
 ENV JOBLIB_TEMP_FOLDER=/tmp
 
-ENV VERSION=7.0.0 \
+ENV VERSION=7.1.0 \
   VERSION_MAJOR=7 \
-  VERSION_MINOR=0 \
+  VERSION_MINOR=1 \
   VERSION_MICRO=0
+
+FROM production AS test
+COPY .circleci/test_image.py .
+COPY CHANGELOG.md .
+
+# Defaults to production as the final stage
+FROM production

README.md

@@ -2,9 +2,7 @@
 
 [![CircleCI](https://circleci.com/gh/civisanalytics/datascience-python/tree/master.svg?style=svg)](https://circleci.com/gh/civisanalytics/datascience-python/tree/master)
 
-This image is created from the official Ubuntu 18.04 Docker image and contains popular Python packages for data science.
-
-If you are reading this README on DockerHub, then the links to files in the GitHub respository will be broken. Please read this documentation from [GitHub](https://github.com/civisanalytics/datascience-python) instead.
+If you are reading this README on DockerHub, then the links to files in the GitHub repository will be broken. Please read this documentation from [GitHub](https://github.com/civisanalytics/datascience-python) instead.
 
 # Introduction
 
@@ -35,7 +33,7 @@ to retrieve a reproducible environment.
 
 Inside the datascience-python Docker image, Python packages are installed in the `root`
 environment. For a full list of included Python libraries, see the
-[environment.yml](environment.yml) file.
+[requirements-core.txt](requirements-core.txt) file.
 
 To start a Docker container from the datascience-python image and
 interact with it from a bash prompt, use
@@ -56,7 +54,7 @@ VERSION_MAJOR
 VERSION_MINOR
 VERSION_MICRO
 ```
-VERSION contains the full version string, e.g. "1.0.3". VERSION_MAJOR,
+VERSION contains the full version string, e.g., "1.0.3". VERSION_MAJOR,
 VERSION_MINOR, and VERSION_MICRO each contain a single integer.
 
 ## Joblib Temporary Files
@@ -68,10 +66,6 @@ default location for staging temporary files to the /tmp directory.
 The normal default is /shm. /shm is a RAM disk which defaults to a 64 MB size
 in Docker containers, too small for typical scientific computing.
 
-# Install a new PyPi Package Dependency
-1. Add new dependency to `requirements-core.txt` file. Include the version
-2. Run script `generate-requirements-full.sh`
-
 # Updating existing PyPi Package Version
 1. Update version of existing package in `requirements-core.txt`
 2. Run script `generate-requirements-full.sh`
@@ -96,7 +90,7 @@ and describe any changes in the [change log](CHANGELOG.md).
 This repo has autobuild enabled. Any PR that is merged to master will
 be built as the `latest` tag on Dockerhub.
 Once you are ready to create a new version, go to the "releases" tab of the repository and click
-"Draft a new release". Github will prompt you to create a new tag, release title, and release
+"Draft a new release". GitHub will prompt you to create a new tag, release title, and release
 description. The tag should use semantic versioning in the form "vX.X.X"; "major.minor.micro".
 The title of the release should be the same as the tag. Include a change log in the release description.
 Once the release is tagged, DockerHub will automatically build three identical containers, with labels

generate-requirements-full.sh

@@ -1,3 +1,4 @@
 #!/bin/bash
-pip install pip-tools
-pip-compile --output-file=requirements-full.txt --pip-args='--prefer-binary' requirements-core.txt
+python --version
+pip install --upgrade pip-tools
+pip-compile --output-file=requirements-full.txt --pip-args='--prefer-binary' --strip-extras --upgrade requirements-core.txt

requirements-core.txt

@@ -1,8 +1,8 @@
-awscli==1.29.5
-boto3==1.28.5
-civis==1.16.1
-numpy==1.25.1
-pandas==2.0.3
-requests==2.31.0
-scikit-learn==1.3.0
-scipy==1.11.1
+awscli==1.32.109
+boto3==1.34.109
+civis==2.0.0
+numpy==1.26.4
+pandas==2.2.2
+requests==2.32.2
+scikit-learn==1.5.0
+scipy==1.13.0

requirements-full.txt

@@ -1,37 +1,37 @@
 #
-# This file is autogenerated by pip-compile with Python 3.11
+# This file is autogenerated by pip-compile with Python 3.12
 # by the following command:
 #
-#    pip-compile --output-file=requirements-full.txt --pip-args='--prefer-binary' requirements-core.txt
+#    pip-compile --output-file=requirements-full.txt --pip-args='--prefer-binary' --strip-extras requirements-core.txt
 #
-attrs==23.1.0
+attrs==23.2.0
     # via
     #   jsonschema
     #   referencing
-awscli==1.29.5
+awscli==1.32.109
     # via -r requirements-core.txt
-boto3==1.28.5
+boto3==1.34.109
     # via -r requirements-core.txt
-botocore==1.31.5
+botocore==1.34.109
     # via
     #   awscli
     #   boto3
     #   s3transfer
-certifi==2023.5.7
+certifi==2024.2.2
     # via requests
-charset-normalizer==3.2.0
+charset-normalizer==3.3.2
     # via requests
-civis==1.16.1
+civis==2.0.0
     # via -r requirements-core.txt
-click==8.1.6
+click==8.1.7
     # via civis
-cloudpickle==2.2.1
+cloudpickle==3.0.0
     # via civis
-colorama==0.4.4
+colorama==0.4.6
     # via awscli
 docutils==0.16
     # via awscli
-idna==3.4
+idna==3.7
     # via requests
 jmespath==1.0.1
     # via
@@ -41,65 +41,65 @@ joblib==1.2.0
     # via
     #   civis
     #   scikit-learn
-jsonref==0.2
+jsonref==1.1.0
     # via civis
-jsonschema==4.18.4
+jsonschema==4.22.0
     # via civis
-jsonschema-specifications==2023.7.1
+jsonschema-specifications==2023.12.1
     # via jsonschema
-numpy==1.25.1
+numpy==1.26.4
     # via
     #   -r requirements-core.txt
     #   pandas
     #   scikit-learn
     #   scipy
-pandas==2.0.3
+pandas==2.2.2
     # via -r requirements-core.txt
-pyasn1==0.5.0
+pyasn1==0.6.0
     # via rsa
-python-dateutil==2.8.2
+python-dateutil==2.9.0.post0
     # via
     #   botocore
     #   pandas
-pytz==2023.3
+pytz==2024.1
     # via pandas
 pyyaml==6.0.1
     # via
     #   awscli
     #   civis
-referencing==0.30.0
+referencing==0.35.1
     # via
     #   jsonschema
     #   jsonschema-specifications
-requests==2.31.0
+requests==2.32.2
     # via
     #   -r requirements-core.txt
     #   civis
-rpds-py==0.9.2
+rpds-py==0.18.1
     # via
     #   jsonschema
     #   referencing
 rsa==4.7.2
     # via awscli
-s3transfer==0.6.1
+s3transfer==0.10.1
     # via
     #   awscli
     #   boto3
-scikit-learn==1.3.0
+scikit-learn==1.5.0
     # via -r requirements-core.txt
-scipy==1.11.1
+scipy==1.13.0
     # via
     #   -r requirements-core.txt
     #   scikit-learn
 six==1.16.0
     # via python-dateutil
-tenacity==8.2.2
+tenacity==8.3.0
     # via civis
-threadpoolctl==3.2.0
+threadpoolctl==3.5.0
     # via scikit-learn
-tzdata==2023.3
+tzdata==2024.1
     # via pandas
-urllib3==1.26.16
+urllib3==2.2.1
     # via
     #   botocore
     #   requests