Skip to content

Commit

Permalink
Mass renaming.
Browse files Browse the repository at this point in the history
  • Loading branch information
Sachaa-Thanasius committed Sep 3, 2024
1 parent b9a260d commit 484557a
Show file tree
Hide file tree
Showing 15 changed files with 201 additions and 226 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
uses: jakebailey/pyright-action@v2
with:
python-version: ${{ matrix.python-version }}
verify-types: deferred
verify-types: defer-imports

test:
runs-on: ${{ matrix.os }}
Expand Down
48 changes: 22 additions & 26 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
========
deferred
defer-imports
========

.. image:: https://github.com/Sachaa-Thanasius/deferred/actions/workflows/ci.yml/badge.svg
:alt: CI Status
:target: https://github.com/Sachaa-Thanasius/deferred/actions/workflows/ci.yml

.. image:: https://img.shields.io/github/license/Sachaa-Thanasius/deferred.svg
.. image:: https://img.shields.io/github/license/Sachaa-Thanasius/defer-imports.svg
:alt: License: MIT
:target: https://opensource.org/licenses/MIT

Expand All @@ -21,7 +17,7 @@ Installation

This can be installed via pip::

python -m pip install git@https://github.com/Sachaa-Thanasius/deferred
python -m pip install git@https://github.com/Sachaa-Thanasius/defer-imports

It can also easily be vendored, as it has zero dependencies and is less than 1,000 lines of code.

Expand All @@ -32,25 +28,25 @@ Usage
Setup
-----

``deferred`` hooks into the Python import system with a path hook. That path hook needs to be registered before code using the import-delaying context manager, ``defer_imports_until_use``, is parsed. To do that, include the following somewhere such that it will be executed before your code:
``defer-imports`` hooks into the Python import system with a path hook. That path hook needs to be registered before code using the import-delaying context manager, ``defer_imports.until_use``, is parsed. To do that, include the following somewhere such that it will be executed before your code:

.. code:: python
import deferred
import defer_imports
deferred.install_defer_import_hook()
defer_imports.install_defer_import_hook()
Example
-------

Assuming the path hook has been registered, you can use the ``defer_imports_until_use`` context manager to decide which imports should be deferred. For instance:
Assuming the path hook has been registered, you can use the ``defer_imports.until_use`` context manager to decide which imports should be deferred. For instance:

.. code:: python
from deferred import defer_imports_until_use
import defer_imports
with defer_imports_until_use:
with defer_imports.until_use:
import inspect
from typing import Final
Expand All @@ -62,7 +58,7 @@ Use Cases

- If imports are necessary to get symbols that are only used within annotations, but such imports would cause import chains.

- The current workaround for this is to perform the problematic imports within ``if typing.TYPE_CHECKING: ...`` blocks and then stringify the fake-imported, nonexistent symbols to prevent NameErrors at runtime; however, resulting annotations are difficult to introspect with standard library introspection tools, since they assume the symbols exist. Using ``with defer_imports_until_use: ...`` in such a circumstance would ensure that the symbols will be imported and saved in the local namespace, but only upon introspection, making the imports non-circular and almost free in most circumstances.
- The current workaround for this is to perform the problematic imports within ``if typing.TYPE_CHECKING: ...`` blocks and then stringify the fake-imported, nonexistent symbols to prevent NameErrors at runtime; however, the resulting annotations raise errors on introspection. Using ``with defer_imports.until_use: ...`` instead would ensure that the symbols will be imported and saved in the local namespace, but only upon introspection, making the imports non-circular and almost free in most circumstances.

- If expensive imports are only necessary for certain code paths that won't always be taken, e.g. in subcommands in CLI tools.

Expand Down Expand Up @@ -104,11 +100,11 @@ How?

The core of this package is quite simple: when import statments are executed, the resulting values are special proxies representing the delayed import, which are then saved in the local namespace with special keys instead of normal string keys. When a user requests the normal string key corresponding to the import, the relevant import is executed and both the special key and the proxy replace themselves with the correct string key and import result. Everything stems from this.

The ``defer_imports_until_used`` context manager is what causes the proxies to be returned by the import statements: it temporarily replaces ``builtins.__import__`` with a version that will give back proxies that store the arguments needed to execute the *actual* import at a later time.
The ``defer_imports.until_use`` context manager is what causes the proxies to be returned by the import statements: it temporarily replaces ``builtins.__import__`` with a version that will give back proxies that store the arguments needed to execute the *actual* import at a later time.

Those proxies don't use those stored ``__import__`` arguments themselves, though; the aforementioned special keys are what use the proxy's stored arguments to trigger the late import. These keys are aware of the namespace, the *dictionary*, they live in, are aware of the proxy they are the key for, and have overriden their ``__eq__`` and ``__hash__`` methods so that they know when they've been queried. In a sense, they're like descriptors, but instead of "owning the dot", they're "owning the brackets". Once such a key has been matched (i.e. someone uses the name of the import), it can use its corresponding proxy's stored arguments to execute the late import and *replace itself and the proxy* in the local namespace. That way, as soon as the name of the deferred import is referenced, all a user sees in the local namespace is a normal string key and the result of the resolved import.

The missing intermediate step is making sure these special proxies are stored with these special keys in the namespace. After all, Python name binding semantics only allow regular strings to be used as variable names/namespace keys; how can this be bypassed? ``deferred``'s answer is a little compile-time instrumentation. When a user calls ``deferred.install_deferred_import_hook()`` to set up the library machinery (see "Setup" above), what they are actually doing is installing an import hook that will modify the code of any given Python file that uses the ``defer_imports_until_use`` context manager. Using AST transformation, it adds a few lines of code around imports within that context manager to reassign the returned proxies to special keys in the local namespace (via ``locals()``).
The missing intermediate step is making sure these special proxies are stored with these special keys in the namespace. After all, Python name binding semantics only allow regular strings to be used as variable names/namespace keys; how can this be bypassed? ``defer-imports``'s answer is a little compile-time instrumentation. When a user calls ``defer_imports.install_deferred_import_hook()`` to set up the library machinery (see "Setup" above), what they are actually doing is installing an import hook that will modify the code of any given Python file that uses the ``defer_imports.until_use`` context manager. Using AST transformation, it adds a few lines of code around imports within that context manager to reassign the returned proxies to special keys in the local namespace (via ``locals()``).

With this methodology, we can avoid using implementation-specific hacks like frame manipulation to modify the locals. We can even avoid changing the contract of ``builtins.__import__``, which specifically says it does not modify the global or local namespaces that are passed into it. We may modify and replace members of it, but at no point do we change its size while within ``__import__`` by removing or adding anything.

Expand All @@ -122,8 +118,8 @@ A bit rough, but there are currently two ways of measuring activation and/or imp

- To prevent bytecode caching from impacting the benchmark, run with `python -B <https://docs.python.org/3/using/cmdline.html#cmdoption-B>`_, which will set ``sys.dont_write_bytecode`` to ``True`` and cause the benchmark script to purge all existing ``__pycache__`` folders in the project directory.
- PyPy is excluded from the benchmark since it takes time to ramp up.
- The cost of registering ``deferred``'s import hook is ignored since that is a one-time startup cost that will hopefully be reduced in time.
- An sample run:
- The cost of registering ``defer-imports``'s import hook is ignored since that is a one-time startup cost that will hopefully be reduced in time.
- An sample run across versions using ``hatch run benchmark:bench``:

(Run once with ``__pycache__`` folders removed and ``sys.dont_write_bytecode=True``):

Expand All @@ -132,28 +128,28 @@ A bit rough, but there are currently two ways of measuring activation and/or imp
============== ======= ========== ===================
CPython 3.9 regular 0.48585s (409.31x)
CPython 3.9 slothy 0.00269s (2.27x)
CPython 3.9 deferred 0.00119s (1.00x)
CPython 3.9 defer-imports 0.00119s (1.00x)
\-\- \-\- \-\- \-\-
CPython 3.10 regular 0.41860s (313.20x)
CPython 3.10 slothy 0.00458s (3.43x)
CPython 3.10 deferred 0.00134s (1.00x)
CPython 3.10 defer-imports 0.00134s (1.00x)
\-\- \-\- \-\- \-\-
CPython 3.11 regular 0.60501s (279.51x)
CPython 3.11 slothy 0.00570s (2.63x)
CPython 3.11 deferred 0.00216s (1.00x)
CPython 3.11 defer-imports 0.00216s (1.00x)
\-\- \-\- \-\- \-\-
CPython 3.12 regular 0.53233s (374.40x)
CPython 3.12 slothy 0.00552s (3.88x)
CPython 3.12 deferred 0.00142s (1.00x)
CPython 3.12 defer-imports 0.00142s (1.00x)
\-\- \-\- \-\- \-\-
CPython 3.13 regular 0.53704s (212.19x)
CPython 3.13 slothy 0.00319s (1.26x)
CPython 3.13 deferred 0.00253s (1.00x)
CPython 3.13 defer-imports 0.00253s (1.00x)
============== ======= ========== ===================

- ``python -m timeit -n 1 -r 1 -- "import deferred"``
- ``python -m timeit -n 1 -r 1 -- "import defer_imports"``

- Substitute ``deferred`` with other modules, e.g. ``slothy``, to compare.
- Substitute ``defer_imports`` with other modules, e.g. ``slothy``, to compare.
- This has great variance, so only value the resulting time relative to another import's time in the same process if possible.


Expand All @@ -162,6 +158,6 @@ Acknowledgements

- All the packages mentioned in "Why?" above, for providing inspiration.
- `PEP 690 <https://peps.python.org/pep-0690/>`_ and its authors, for pushing lazy imports to the point of almost being accepted as a core part of CPython's import system.
- Jelle Zijlstra, for so easily creating and sharing a `sample implementation <https://gist.github.com/JelleZijlstra/23c01ceb35d1bc8f335128f59a32db4c>`_ that ``slothy`` and ``deferred`` are based on.
- Jelle Zijlstra, for so easily creating and sharing a `sample implementation <https://gist.github.com/JelleZijlstra/23c01ceb35d1bc8f335128f59a32db4c>`_ that ``slothy`` and ``defer-imports`` are based on.
- `slothy <https://github.com/bswck/slothy>`_, for being a major reference and inspiration for this project.
- Sinbad, for all his feedback.
14 changes: 7 additions & 7 deletions benchmark/bench_samples.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# pyright: reportUnusedImport=none
"""Simple benchark script for comparing the import time of the Python standard library when using regular imports,
deferred-influence imports, and slothy-influenced imports.
defer_imports-influence imports, and slothy-influenced imports.
The sample scripts being imported are generated with benchmark/generate_samples.py.
"""
Expand All @@ -10,7 +10,7 @@
import time
from pathlib import Path

import deferred
import defer_imports


class CatchTime:
Expand Down Expand Up @@ -44,13 +44,13 @@ def bench_regular() -> float:
return ct.elapsed


def bench_deferred() -> float:
deferred.install_defer_import_hook()
def bench_defer_imports() -> float:
defer_imports.install_defer_import_hook()

with CatchTime() as ct:
import benchmark.sample_deferred
import benchmark.sample_defer_imports

deferred.uninstall_defer_import_hook()
defer_imports.uninstall_defer_import_hook()
return ct.elapsed


Expand All @@ -63,7 +63,7 @@ def bench_slothy() -> float:
BENCH_FUNCS = {
"regular": bench_regular,
"slothy": bench_slothy,
"deferred": bench_deferred,
"defer_imports": bench_defer_imports,
}


Expand Down
16 changes: 8 additions & 8 deletions benchmark/generate_samples.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Generate sample scripts with the same set of imports but influenced by different libraries, e.g. deferred."""
"""Generate sample scripts with the same set of imports but influenced by different libraries, e.g. defer_imports."""

from pathlib import Path

Expand Down Expand Up @@ -563,21 +563,21 @@ def main() -> None:
regular_contents = "\n".join((PYRIGHT_IGNORE_DIRECTIVES, GENERATED_BY_COMMENT, STDLIB_IMPORTS))
regular_path.write_text(regular_contents, encoding="utf-8")

# deferred-instrumented and deferred-hooked imports
deferred_path = bench_path / "sample_deferred.py"
deferred_contents = (
# defer_imports-instrumented and defer_imports-hooked imports
defer_imports_path = bench_path / "sample_defer_imports.py"
defer_imports_contents = (
f"{PYRIGHT_IGNORE_DIRECTIVES}\n"
f"{GENERATED_BY_COMMENT}\n"
"from deferred import defer_imports_until_use\n"
"import defer_imports\n"
"\n"
"\n"
"with defer_imports_until_use:\n"
"with defer_imports.until_use:\n"
f"{INDENTED_STDLIB_IMPORTS}"
)
deferred_path.write_text(deferred_contents, encoding="utf-8")
defer_imports_path.write_text(defer_imports_contents, encoding="utf-8")

tests_path = Path().resolve() / "tests" / "stdlib_imports.py"
tests_path.write_text(deferred_contents, encoding="utf-8")
tests_path.write_text(defer_imports_contents, encoding="utf-8")

# slothy-hooked imports
slothy_path = bench_path / "sample_slothy.py"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# pyright: reportUnusedImport=none, reportMissingTypeStubs=none
# Generated by benchmark/generate_samples.py
from deferred import defer_imports_until_use
import defer_imports


with defer_imports_until_use:
with defer_imports.until_use:
import __future__

# import _bootlocale # Doesn't exist on 3.11 on Windows
Expand Down
32 changes: 15 additions & 17 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,12 @@ requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "deferred"
name = "defer-imports"
description = "Lazy imports with regular syntax in pure Python."
requires-python = ">=3.9"
license = "MIT"
readme = { file = "README.rst", content-type = "text/x-rst" }
authors = [
{ name = "Sachaa-Thanasius", email = "[email protected]" },
]
authors = [{ name = "Sachaa-Thanasius" }]
classifiers = [
"Development Status :: 3 - Alpha",
"Natural Language :: English",
Expand All @@ -31,21 +29,21 @@ classifiers = [
dynamic = ["version"]

[tool.hatch.version]
path = "src/deferred/_core.py"
path = "src/defer_imports/_core.py"

[project.optional-dependencies]
benchmark = ["slothy"]
test = ["pytest"]
cov = ["deferred[test]", "coverage", "covdefaults"]
dev = ["deferred[benchmark,cov]", "pre-commit"]
cov = ["defer-imports[test]", "coverage", "covdefaults"]
dev = ["defer-imports[benchmark,cov]", "pre-commit"]

[project.urls]
Documentation = "https://github.com/Sachaa-Thanasius/deferred#readme"
Issues = "https://github.com/Sachaa-Thanasius/deferred/issues"
Source = "https://github.com/Sachaa-Thanasius/deferred"
Documentation = "https://github.com/Sachaa-Thanasius/defer-imports#readme"
Issues = "https://github.com/Sachaa-Thanasius/defer-imports/issues"
Source = "https://github.com/Sachaa-Thanasius/defer-imports"

[tool.hatch.build.targets.wheel]
packages = ["src/deferred"]
packages = ["src/defer_imports"]


# -------- Benchmark config
Expand Down Expand Up @@ -77,11 +75,11 @@ addopts = [
]

[tool.coverage.paths]
deferred = ["src"]
defer_imports = ["src"]

[tool.coverage.run]
plugins = ["covdefaults"]
source = ["deferred", "tests"]
source = ["defer_imports", "tests"]

[tool.coverage.report]
# It's a work in progress.
Expand Down Expand Up @@ -157,7 +155,7 @@ extend-ignore = [
unfixable = [
"ERA", # Prevent unlikely erroneous deletion.
]
typing-modules = ["deferred._typing"]
typing-modules = ["defer_imports._typing"]

[tool.ruff.lint.isort]
lines-after-imports = 2
Expand All @@ -173,10 +171,10 @@ keep-runtime-typing = true
"F403",
"F405",
]
"src/deferred/_core.py" = [
"src/defer_imports/_core.py" = [
"A002", # Allow some shadowing of builtins by parameter names.
]
"src/deferred/_typing.py" = [
"src/defer_imports/_typing.py" = [
"F822", # __all__ has names that are only provided by module-level __getattr__.
]

Expand All @@ -201,7 +199,7 @@ keep-runtime-typing = true
# -------- Type-checker config

[tool.pyright]
include = ["src/deferred", "tests"]
include = ["src/defer_imports", "tests"]
pythonVersion = "3.9"
pythonPlatform = "All"
typeCheckingMode = "strict"
Expand Down
4 changes: 2 additions & 2 deletions src/deferred/__init__.py → src/defer_imports/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@
"""

from ._console import DeferredInteractiveConsole
from ._core import __version__, defer_imports_until_use, install_defer_import_hook, uninstall_defer_import_hook
from ._core import __version__, install_defer_import_hook, uninstall_defer_import_hook, until_use


__all__ = (
"__version__",
"defer_imports_until_use",
"install_defer_import_hook",
"uninstall_defer_import_hook",
"until_use",
"DeferredInteractiveConsole",
)
File renamed without changes.
4 changes: 2 additions & 2 deletions src/deferred/_console.py → src/defer_imports/_console.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@


class DeferredInteractiveConsole(InteractiveConsole):
"""An emulator of the interactive Python interpreter, but with deferred's compile-time hook baked in to ensure that
defer_imports_until_use works as intended directly in the console.
"""An emulator of the interactive Python interpreter, but with defer_import's compile-time hook baked in to ensure that
defer_imports.until_use works as intended directly in the console.
"""

def __init__(self) -> None:
Expand Down
Loading

0 comments on commit 484557a

Please sign in to comment.