v0.4.0

This release contains the nnbench CLI as a headline feature for running benchmarking workloads as part of scripts and CML pipelines.

It also contains a large refactor of core abstractions such as the benchmark runners and reporters. In particular, the nnbench.FileReporter got support for writing files with fsspec, making it possible to stream records to all major cloud storage providers among others.

What's Changed

• (fix) Change default == value to is by @maxmynter in #139
• Add nnbench video to readme by @maxmynter in #144
• Safe setting of CPU Info by @maxmynter in #146
• Bump oldest supported Python to 3.10 by @nicholasjng in #147
• Switch project to uv lockfile by @nicholasjng in #148
• Remove requirement files, strict version pins from examples by @nicholasjng in #149
• Remove unused io module stub by @nicholasjng in #150
• Add CLI entrypoints for module and program invocation by @nicholasjng in #152
• Run pre-commit autoupdate by @nicholasjng in #153
• Add Python 3.13 tests to CI by @nicholasjng in #155
• refactor: Remove context class, back to raw dictionaries by @nicholasjng in #154
• Remove setuptools_scm in favor of explicit version management by @nicholasjng in #156
• Remove transform facility and examples by @nicholasjng in #161
• Remove context processing options by @nicholasjng in #160
• Add Benchmark.to_list() and to_json() to improve record IO by @nicholasjng in #162
• Remove compression, support file reporting and context passing on the command line by @nicholasjng in #163
• Add nnbench compare subcommand, comparison function by @nicholasjng in #164
• Refactor console reporter by @nicholasjng in #165
• Add --version switch to print version by @nicholasjng in #166
• Inline boolean option pairs, change subcommand section by @nicholasjng in #167
• Add run name to benchmark record by @nicholasjng in #171
• Simplify JSON representation computation facility by @nicholasjng in #172
• Add a fixture facility to nnbench by @nicholasjng in #173
• Remove typechecks from benchmark runner by @nicholasjng in #174
• Add TOML config parsing stub by @nicholasjng in #175
• Add artifact outputs to gitignore by @nicholasjng in #176
• Add API docs for all direct nnbench submodules by @nicholasjng in #178
• Add nnbench CLI guide by @nicholasjng in #179
• Make arguments optional on ContextProviderDef by @nicholasjng in #181
• Add fixture usage guide document by @nicholasjng in #182
• Fix README lightning demo example by @nicholasjng in #183
• Update code examples, fix parameter hydration bug by @nicholasjng in #184
• Refactor jsonifier to be taken as an argument to BenchmarkRunner.run() by @nicholasjng in #185

Full Changelog: v0.3.0...v0.4.0

v0.3.0

This release contains the Memo class for better memory efficiency when benchmarking multiple models or datasets with large memory footprint in succession.

It also introduces Transforms as a way to encapsulate processing of records for multiple purposes, like generating statistics or serialization/deserialization.

What's Changed

• Example Guide: Benchmark on Saved Models by @maxmynter in #92
• Add a lightning demo to the README.md by @nicholasjng in #98
• Hotfix: Adjust snippet paths by @nicholasjng in #102
• Add Parameters to Benchmarks by @maxmynter in #103
• Run pre-commit autoupdate, bump dependencies by @nicholasjng in #104
• Implement a FSspec based ArtifactLoader by @maxmynter in #100
• Add Artifacts guide by @maxmynter in #107
• Remove the ArtifactCollection class by @maxmynter in #109
• Inline the _cleanup function for the FilePathArtifactLoader. by @maxmynter in #111
• Unique keys by @maxmynter in #112
• functools.partialize the nnbench.parametrize decorators. by @maxmynter in #116
• Add transform proposal by @nicholasjng in #113
• Make Transform an abstract base class, delete iapply method ... by @nicholasjng in #118
• Check loaded module names before importing a new one from a file by @nicholasjng in #119
• Rollforward of lakefs-spec doc changes by @nicholasjng in #123
• Add transform submodule, parameter compression transform by @nicholasjng in #124
• Feature: Thunks as an alternative to artifacts by @nicholasjng in #120
• Add new memoization guide by @nicholasjng in #129
• State injection into setup and teardown by @maxmynter in #127
• Add global memo cache and integrate with the setUp and teardown injection by @maxmynter in #130
• Update dev requirements, add Memray artifacts to gitignore by @nicholasjng in #136
• Organize nnbench.types.types into different files by @maxmynter in #135
• Implement memo garbage collection by @nicholasjng in #137
• Update HuggingFace example to use memoization by @nicholasjng in #133

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Summary

This release makes a wide variety of benchmarking functionality available: Contexts for easier metadata organization, providers for basic information, file IO for record persistence on disk, and more.

It also adds new guides and code examples on how to benchmark code easily in a variety of scenarios, such as in orchestrator workflows and with duckDB.

What's Changed

• Add git info and Python package info providers by @nicholasjng in #64
• Group context structs under top-level semantic keys by @nicholasjng in #65
• Simplify nullcols helper, flatten nested context by @nicholasjng in #68
• Fix runner collection by checking file path first by @nicholasjng in #70
• Change nnbench.reporter file to submodule, add base reporter by @nicholasjng in #71
• Inline console reporter into BenchmarkReporter.display() by @nicholasjng in #76
• Orchestrator examples by @maxmynter in #73
• Improve some aspects on benchmark reporters by @nicholasjng in #77
• Add barebones duckDB reporter by @nicholasjng in #75
• Add BenchmarkContext Class by @maxmynter in #79
• Improve file drivers, add dict roundtrip methods to BenchmarkRecord by @nicholasjng in #82
• Unify batching in file drivers, remove read/write stubs on base reporter by @nicholasjng in #83
• Add compression algorithms facility to file IO by @nicholasjng in #84
• Streamlit Integration by @maxmynter in #81
• Add BQ example to showcase cloud streaming of benchmark records by @nicholasjng in #85
• Add parquet, ndjson drivers, change registration mechanisms by @nicholasjng in #87
• Revert file extension coercion, add raise statement for missing driver by @nicholasjng in #88
• Switch author and maintainer description in example .toml's by @maxmynter in #89
• Restructure test suite and add file IO roundtrip tests by @nicholasjng in #90
• Add duckDB example by @nicholasjng in #91
• Run pre-commit autoupdate, upgrade requirement lockfiles by @nicholasjng in #93

Full Changelog: v0.1.0...v0.2.0

v0.1.0

This marks the first stable release of nnbench. From now on, the package is available for installation on PyPI using standard Python dependency management tools like pip and poetry.

Contents

• Definition and parametrization of benchmarks via decorators
• Collection and running in a benchmark runner class
• Result reporting via a benchmark reporter class (currently, only tabular console output is supported.)
• Documentation including quickstart, a more advanced example, and an API reference.

Breaking changes

None.

Deprecations

None.

v0.1.0-rc1

This is a test PyPI release.