Extend the documentation. (#11)

Author: tobiasraabe

.gitignore

@@ -9,6 +9,7 @@ coverage.*
 .vscode
 _build
 __pycache__
+_generated
 
 *.egg-info
 

.pre-commit-config.yaml

@@ -32,7 +32,6 @@ repos:
     rev: 3.8.3
     hooks:
     -   id: flake8
-        types: [python]
         additional_dependencies: [
             flake8-alfred,
             flake8-bugbear,
@@ -55,12 +54,10 @@ repos:
       name: doc8
       description: This hook runs doc8 for linting docs
       entry: python -m doc8
-      language: python
-      files: \.rst$
-- repo: https://github.com/codespell-project/codespell
-  rev: v1.17.1
-  hooks:
-    - id: codespell
+-   repo: https://github.com/codespell-project/codespell
+    rev: v1.17.1
+    hooks:
+    -   id: codespell
 -   repo: meta
     hooks:
     -   id: check-hooks-apply

README.rst

@@ -99,7 +99,7 @@ To execute the task, type the following command on the command-line
 Documentation
 -------------
 
-The documentation can be found under https://pytask-dev.readthedocs.io/en/latest and
+The documentation can be found under https://pytask-dev.readthedocs.io/en/latest with
 tutorials and explanations.
 
 

docs/_static/css/custom.css

@@ -3,25 +3,6 @@ div.prompt {
     display: none;
 }
 
-/* General options */
-
-h1, h2, h3, h4, h5, h6 {
-    color: #547482 !important;
-}
-
-a {
-    color: #547482;
-}
-
-a.headerlink {
-    color: #C87259 !important;
-}
-
-a.headerlink:hover {
-    color: #fff !important;
-    background-color: #C87259 !important;
-}
-
 /* Getting started index page */
 
 .intro-card {
@@ -70,7 +51,7 @@ a#index-link {
 
 .gs-torefguide a:hover {
     margin-left: 5px;
-    color: grey;
+    color: #333;
     text-decoration: none;
     border-bottom: 1px solid #b2ff80f3;
     box-shadow: 0px -10px 0px #b2ff80f3 inset;

docs/changes.rst

@@ -5,6 +5,15 @@ This is a record of all past pytask releases and what went into them in reverse
 chronological order. Releases follow `semantic versioning <https://semver.org/>`_ and
 all releases are available on `Anaconda.org <https://anaconda.org/pytask/pytask>`_.
 
+
+0.0.5 - 2020-xx-xx
+------------------
+
+- :gh:`10` turns parametrization into a plugin.
+- :gh:`11` extends the documentation.
+- :gh:`12` replaces ``pytest.mark`` with ``pytask.mark``.
+
+
 0.0.4 - 2020-07-22
 ------------------
 

docs/conf.py

@@ -31,10 +31,15 @@
 extensions = [
     "IPython.sphinxext.ipython_console_highlighting",
     "IPython.sphinxext.ipython_directive",
-    "numpydoc",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
     "sphinx.ext.extlinks",
     "sphinx.ext.intersphinx",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
     "sphinx_copybutton",
+    "autoapi.extension",
+    "sphinx_autodoc_typehints",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -46,7 +51,6 @@
 # html_extra_path.
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
-
 # -- Extensions configuration ----------------------------------------------------------
 
 # Configuration for autodoc
@@ -64,11 +68,14 @@
 
 intersphinx_mapping = {
     "python": ("https://docs.python.org/3.8", None),
+    "click": ("https://click.palletsprojects.com/en/7.x", None),
+    "pluggy": ("https://pluggy.readthedocs.io/en/latest", None),
 }
 
-# Configuration for numpydoc
-numpydoc_xref_param_type = True
-numpydoc_xref_ignore = {"type", "optional", "default", "of"}
+# Configuration for autoapi
+autoapi_type = "python"
+autoapi_dirs = ["../src/pytask"]
+autoapi_keep_files = False
 
 
 # -- Options for HTML output -----------------------------------------------------------

docs/explanations/design.rst

@@ -0,0 +1,13 @@
+Design
+======
+
+The design of pytask has some key objectives.
+
+1. The interface must be simple, easy-to-learn, and may have synergies with pytest. It
+   is important that even users without a strong background in computer science are able
+   to use pytask.
+
+2. pytask must be easily extensible via plugins. Developers of pytask are naturally
+   unaware of all the possible applications of a build system. Thus, they must focus on
+   the host application and the design of the entry-points. This will also reduce the
+   amount of maintenance.

docs/explanations/index.rst

@@ -1,11 +1,11 @@
 Explanations
 ============
 
-.. note::
+Explanations provide background information and historical context on pytask and build
+systems in general as well as its design.
 
-    Unfortunately, this section is still under development. If you want to contribute to
-    the documentation, please file an issue to start a conversation or create a pull
-    request.
+.. toctree::
+   :maxdepth: 1
 
-Explanations offer background information on the software, discuss design decisions and
-alternative implementations.
+   why_another_build_system
+   design

docs/explanations/why_another_build_system.rst

@@ -0,0 +1,91 @@
+Why another build system?
+=========================
+
+There are a lot of build systems out there with existing communities who accumulated a
+lot of experience over time. So why go through the hassle of creating another build
+system?
+
+One reason was pure curiosity. `Waf <https://waf.io>`_ has been used for some time as a
+build system for research projects. Although, it has several limitations (difficult
+interface, no debugging, cryptic error messages, a bus factor of one), it is also a
+mature library. At some point annoyance won over comfort and I (`@tobiasraabe
+<https://github.com/tobiasraabe>`_) started to write pytask.
+
+Another reason is that pytask is created having a particular audience in mind. Many
+researchers are not computer scientists first, but acquired some programming skills
+throughout their careers. This means a build system must be extremely user-friendly or
+provide a `steep learning curve <https://english.stackexchange.com/a/6226>`_, because it
+is only a tool. Since pytask resembles pytest in some ways, users have an easier time
+switching to pytask and feel more comfortable.
+
+The third reason is that pytest seems to provide the ideal architecture for a build
+system. Its plugin-based design allows for customization at every level. A build system
+is a tool which can be deployed in many different environments whose requirements are
+not foreseeable by the developer. If it is easy for users / developers to write plugins
+which extend the functionality of pytask it is more valuable. If there is any question
+whether pytest's architecture is really suited for this, one should look at the success
+of pytest, its wide-spread adoption, and its over 800 plugins (even if most of them
+might be dead).
+
+
+Alternatives
+------------
+
+Here is a list of other build systems and their advantages and disadvantages compared to
+pytask. The list helps to define pytask's niche and to collect new ideas to improve
+pytask.
+
+
+`snakemake <https://github.com/snakemake/snakemake>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pros
+
+- Mature library.
+- Is the build system for some research projects.
+- Can scale to clusters and use Docker images.
+- Supports Python and R.
+
+Cons
+
+- Need to learn snakemake's syntax which is a mixture of Make and Python.
+- Seems to have no plugin system.
+
+
+`Waf <https://waf.io>`_
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Pros
+
+- Mature library.
+- Can be extended.
+
+Cons
+
+- Focus on compiling binaries.
+- Bus factor of 1.
+
+
+`Scons <https://github.com/SCons/scons>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pros
+
+- Mature library.
+
+Cons
+
+- Seems to have no plugin system.
+
+
+`cook <https://github.com/jachris/cook>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pros
+
+- Still simple, maybe useful for some quick inspirations.
+
+Cons
+
+- Development is paused.
+- No plugin system, but extensible interfaces.

docs/glossary.rst

@@ -3,7 +3,7 @@ Glossary
 
 .. glossary::
 
-    Build system
+    build system
         A build system is a tool for developing software which manages the process from
         compiling the source code to binary code, packaging the binary code and running
         tests.
@@ -13,5 +13,27 @@ Glossary
 
     DAG
         A `directed acyclic graph (DAG) <https://en.wikipedia.org/wiki/
-        Directed_acyclic_graph>`_  is a graph with a finite amount of vertices and edges
+        Directed_acyclic_graph>`_  is a graph with a finite amount of nodes and edges
         which are connected such that no circles exist.
+
+    host
+        The program which offers extensibility via entry-points.
+
+    entry-point
+        Access points for plugins in the host program. At an entry-point, the host sends
+        a message which can be processed by plugins. Then, the plugins may respond.
+
+    plugin
+        A plugin is a software which changes the behavior of the host program by
+        processing the message send by the host at an entry-point. A plugin can consist
+        of one or more :term:`hook implementations <hook implementation>`.
+
+    hooking
+        See the reference guide on :ref:`pluggy` or the more general explanation on
+        `Wikipedia <https://en.wikipedia.org/wiki/Hooking>`_.
+
+    hook specification
+        Another term for :term:`entry-point` when talking about hooking and pluggy.
+
+    hook implementation
+        A part of the plugin which intercepts the message at one specific entry-point.

docs/how_to_guides/how_to_write_a_plugin.rst

@@ -0,0 +1,81 @@
+How to write a plugin
+=====================
+
+Since pytask is built using pluggy, it is easily extensible. In this section, you will
+learn some key concepts you need to know to write a plugin. It won't deal with pluggy in
+detail, but if you are interested feel free to read :ref:`pluggy`. A quick look at the
+first paragraphs might be useful nonetheless.
+
+
+Preparation
+-----------
+
+Before you start implementing your plugin, it makes sense to go through the following
+steps.
+
+- Check whether there exist plugins which offer similar functionality. For example, many
+  plugins provide convenient interfaces to run another program with inputs via the
+  command line. Naturally, there is a lot of overlap in the structure of the program and
+  even the the test battery. Finding the right template may save you a lot of time.
+
+- Make a list of hooks you want to implement. Think about how this plugin relates to
+  functionality defined in pytask and other plugins. Maybe skim the documentation on
+  :ref:`pluggy` to see whether there is advanced pattern
+  which makes your implementation easier.
+
+- You may file an issue on `Github <https://github.com/pytask-dev/pytask>`_ and propose
+  your plugin to get feedback from others. Your proposal should be concise, should
+  explain which problem is solved and how.
+
+
+Writing your plugin
+-------------------
+
+Set up the setuptools entry-point
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+pytask discovers plugins via ``setuptools`` entry-points. This is specified in
+``setup.py``. See the following example.
+
+.. code-block:: python
+
+    # Content of setup.py
+
+    from setuptools import find_packages
+    from setuptools import setup
+
+    setup(
+        name="pytask-plugin",
+        version="0.0.1",
+        entry_points={"pytask": ["pytask_plugin = pytask_plugin.plugin"]},
+        # PyPI classifier for pytask plugins
+        classifiers=["Framework :: pytask"],
+    )
+
+The entry-point for pytask is also called ``"pytask"`` and you point to a module called
+``pytask_plugin.plugin``.
+
+
+plugin.py
+~~~~~~~~~
+
+``plugin.py`` is the main module in your package. You can put all of your hook
+implementations in this module, but it is recommended imitate the structure of pytask
+and its modules. For example, all hook implementations which change the configuration
+should be implemented in ``pytask_plugin.config``.
+
+If you follow the recommendations, the only content in ``plugin.py`` is a single hook
+implementation which registers other hook implementations of your plugin. The following
+example registers all hooks implemented in ``config.py``.
+
+.. code-block:: python
+
+    # Content of plugin.py
+
+    import pytask
+    from pytask_plugin import cli
+
+
+    @pytask.hookimpl
+    def pytask_add_hooks(pm):
+        pm.register(cli)

docs/how_to_guides/index.rst

@@ -1,11 +1,11 @@
 How-to Guides
 =============
 
-.. note::
-
-    Unfortunately, this section is still under development. If you want to contribute to
-    the documentation, please file an issue to start a conversation or create a pull
-    request.
-
 How-to guides provide detailed explanations on how to accomplish specific tasks with
 pytask.
+
+
+.. toctree::
+   :maxdepth: 1
+
+   how_to_write_a_plugin