Initialize repo and add simple memory tests (#1)

* add initial scaffold

* add pre-commit checks workflow

* update linting, add pipeline and runner modules

* linting

* move dockerfile, add a baseline test, pipeline cfg

* rename / typo

* readd dockerfile

* Update pyproject.toml

* update pipeline

* move dockerfile again

* adjust pipeline for multiple input

* linting

* move test; add fire; adjust for multiple input

* add pandas; add debug_leak test; linting cfg

* remove pandas; rename basic tests

* add callbacks to gc test

* update docs

* updates for linting

* update to show the stdout from container

* change dockerfile path

* move to solely list-based execution

* avoid the use of runner env group with tests

* add poethepoet workflow for tests

* remove no longer used union

* remove print, adjust language

* simplify the prints for async ops

* add github actions workflow for running tests

* move to main for running tests

* add guidance and overview in readme

* add python fire mentions

* set the platform of the container

* update naming and docs

* typo

* add poethepoet as a runner dependency

* bandit skip tests for assert linting

* update tests with asserts for garbage

* add debug config; add lines for visual clarity

* expand readme with some definitions + new sections

* add memory allocation definitions

* assumption about python memory

* add more detail to python garbage collection

* reorganizing and adding test module documentation

* add extra cli arg capabilities

* reorganize for better flow; specify allocators

* further details on allocators

* add diagrams

* additional diagrams, specificity

* add blocks content

* Update README.md

* modifications; adding reference counting guide

* add analogy; memray; begin scalene content

* add scalene content

* remove content and images ported to blog post

.github/workflows/pre-commit-checks.yaml

@@ -0,0 +1,19 @@
+---
+# used for running pre-commit checks
+name: pre-commit checks
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  run_pre_commit_tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - uses: pre-commit/[email protected]

.github/workflows/run-memory-tests.yaml

@@ -0,0 +1,27 @@
+---
+# used for running memory checks through GitHub Actions for open observability
+name: run memory tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  run_memory_tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Setup poetry and poethepoet
+        run: |
+          python -m pip install poetry
+      - name: Install poetry runner env
+        run: |
+          poetry install --with runner -v --no-interaction
+      - name: Run memory tests through Dagger pipeline
+        run: |
+          poetry run poe run_all_tests

.gitignore

@@ -158,3 +158,6 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# customizations
+.python-version

.pre-commit-config.yaml

@@ -0,0 +1,67 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+default_language_version:
+  python: python3.11
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-added-large-files
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.6
+    hooks:
+      - id: codespell
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.17
+    hooks:
+      - id: mdformat
+  - repo: https://github.com/hadolint/hadolint
+    rev: v2.12.1-beta
+    hooks:
+      - id: hadolint-docker
+  - repo: https://github.com/rhysd/actionlint
+    rev: v1.6.26
+    hooks:
+      - id: actionlint-docker
+  - repo: https://github.com/adrienverge/yamllint
+    rev: v1.33.0
+    hooks:
+      - id: yamllint
+  - repo: https://github.com/psf/black
+    rev: 23.12.1
+    hooks:
+      - id: black
+  - repo: https://github.com/PyCQA/bandit
+    rev: 1.7.6
+    hooks:
+      - id: bandit
+        args: ["-c", "pyproject.toml"]
+        additional_dependencies: ["bandit[toml]"]
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+  - repo: https://github.com/jendrikseipp/vulture
+    rev: v2.10
+    hooks:
+      - id: vulture
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.8.0
+    hooks:
+      - id: mypy
+  - repo: https://github.com/PyCQA/pylint
+    rev: v3.0.3
+    hooks:
+      - id: pylint
+        name: pylint
+        entry: pylint
+        language: python
+        types: [python]
+        # note: we install additional dependencies for
+        # pylint due to it's dynamic analysis capabilities
+        # and the way pre-commit references virtual environments.
+        additional_dependencies:
+          - "dagger-io"
+          - "fire"

README.md

@@ -1,2 +1,85 @@
-# demo-python-memory-reference-accounting
-Demonstrating Python memory allocator reference counting and debug
+# Demonstrating Python Memory Allocator Reference Counting (and Related)
+
+Demonstrating Python memory allocator reference counting and debug.
+The goal of this repository is to help demonstrate Python memory allocation using source controlled code.
+Work for this project was originally inspired by [cytomining/CytoTable#75](https://github.com/cytomining/CytoTable/issues/75) which explores related topics.
+
+## Development
+
+The following are suggested steps to get started with development for this project.
+
+1. (Suggested) Install Python from [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation) (or another way).
+1. [Install Poetry](https://python-poetry.org/docs/#installation)
+1. [Install Docker Desktop](https://www.docker.com/products/docker-desktop/)
+1. Run Poe the Poet workflow(s): e.g. `poetry run poe run_all_tests`
+   _(Poe the Poet is installed as a Poetry env dependency for the `runner` group)_
+
+## Project outline
+
+```mermaid
+---
+title: Project Overview
+---
+flowchart LR
+
+pyproject["pyproject.toml"]
+poeworkflow["poethepoet\nworkflows"]
+dockerfile["Dockerfile"]
+subgraph test_container["test container"]
+    subgraph tests
+        test_env["Python environment\n(for tests)"]
+        test_modules["Test module(s)"]
+    end
+end
+subgraph runner
+    runner_env["Python environment\n(for runner)"]
+    runner_cli["Runner CLI\n(Python Fire)"]
+    pipeline["Container pipeline\n(through Dagger)"]
+end
+
+
+pyproject --> |defines| test_env
+pyproject --> |defines| runner_env
+
+test_env --> |runs| test_modules
+test_modules -.-> |display\nresults through| pipeline
+runner_env --> |enables| runner_cli
+runner_cli --> |runs| pipeline
+
+dockerfile -.-> |defines\ncontainer for| pipeline
+
+pipeline --> |executes| test_container
+
+pyproject --> poeworkflow
+poeworkflow -.-> |declarative\nworkflows for| runner
+```
+
+See above for a quick overview of project components and their relationship.
+[Poetry](https://python-poetry.org/docs/) is used to define Python environment dependencies within [dependency groups](https://python-poetry.org/docs/master/managing-dependencies/#dependency-groups) in a `pyproject.toml` file.
+Declarative [Poe the Poet tasks](https://poethepoet.natn.io/index.html) may also be found in the same `pyproject.toml` file to help define reproducible workflows.
+A "runner" command-line interface (CLI) is provided through [Python Fire](https://github.com/google/python-fire) to help enable the use of the container-based pipelines.
+Container-based pipelines are provided through [Dagger's Python SDK](https://docs.dagger.io/sdk/python/) to help isolate potential OS-specific distinctions for memory allocation work in Python.
+Testing workflows are designed to run "locally" within a developer's environment (for example, leveraging [pyenv](https://github.com/pyenv/pyenv), [poetry](https://python-poetry.org/docs/), and [Docker Desktop](https://www.docker.com/products/docker-desktop/)) or within [GitHub Actions images](https://github.com/actions/runner-images) (`dagger-io` installs the necessary dependencies).
+
+## Test Modules
+
+This project focuses on leveraging Python memory observability tools to illustrate what happens as code is executed.
+See the `src/pymaccounter/tests` folder for a full list of test modules.
+Each test module includes a description of what it tests and expects in a [docstring](https://docs.python.org/3/glossary.html#term-docstring) near the top of the file.
+
+### Running Test Modules
+
+Test modules may be executed individually or in groups.
+Test modules are provided in a list to be run by containerized pipelines.
+Each test provided in this way is run in an isolated container instance.
+
+In addition to test module specification, a test module base directory and debug mode may also be specified.
+The test module base directory is where the container pipeline will look for test modules listed by name in the list.
+Debug mode may be used to view container pipeline debug log messages.
+
+See the following examples for more details on the suggested way to run tests through this project.
+
+- Individual test: `poetry run python src/pymaccounter/runner.py '["test_baseline.py"]'`
+- Multiple tests: `poetry run python src/pymaccounter/runner.py '["test_baseline.py", "test_multiply_gc.collect.py"]'`
+- Individual test with debug mode: `poetry run python src/pymaccounter/runner.py '["test_baseline.py"]' --debug True`
+- Individual test with non-default base test directory specification: `poetry run python src/pymaccounter/runner.py '["test_baseline.py"]' --test_dir 'src/another_test_dir'`