Merge branch 'release/v0.1.8'

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.1.7
+current_version = 0.1.8
 commit = False
 tag = False
 allow_dirty = False

.gitignore

@@ -12,9 +12,18 @@ docs/sensai
 dist
 *.egg-info*
 /temp
+/temp*.py
 /default_console.py
 /build
 /git-split.sh
 /.pytest_cache
 data
 .coverage
+/*.env
+/docs-build
+/code.sh
+/lightning_logs
+*.code-workspace
+/TODO.txt
+notebooks/temp.ipynb
+/docs/build

README-dev.md

@@ -1,15 +1,59 @@
+# Development Environment
+
+This section explains the steps required to set up an environment in order to develop sensAI further.
+
+## Clone Large Files
+
+Clone the full repo, including large files using [git LFS](https://git-lfs.github.com):
+
+    git lfs pull
+
+This adds, in particular, data that is used in notebooks.
+
+## Create the Python Virtual Environment
+
+Use conda to set up the Python environment:
+
+    conda env create -f environment.py
+
+Solving the environment may take several minutes (but should ultimately work).
+
+NOTE: versions are mostly unpinned in the environment specification, because this facilitates conda dependency resolution. Also, sensAI is intended to be compatible with all (newer) versions of the dependencies. If it isn't, we need to specify  an upper version bound in `setup.py` (where it matters the most) as well as in `environment.yml`. Compatibility with old (pinned) versions and the latest versions is tested in the tox build (see below).
+
 # Build and Test Pipeline
 
 The tests and docs build are executed via **tox** in several environments:
 * `py`: the "regular" test environment, where we test against the pinned dependencies which we also use for development (by explicitly including `requirements.txt` with the pinned versions; this is also the environment in which we test the execution of notebooks
 * `py_latest_dependencies`: the environment where we use the latest versions of all dependencies (except where we have identified an incompatibility; see `setup.py` definitions `DEPS_VERSION_LOWER_BOUND` and `DEPS_VERSION_UPPER_BOUND_EXCLUSIVE`); by not including `requirements.txt`, we depend on the latest admissible versions according to `setup.py`
-* `docs`: the environment in which docs are built via sphinx (by executing `build_scripts/update_docs.py`)
+* `docs`: the environment in which docs are built via sphinx 
 
 ## Docs Build
 
-Docs are automatically created, all .rst files are auto-generated; only `index.rst` is manually defined.
+Docs are automatically created during the GitHub build via tox.
+
+All .rst files are auto-generated (by `build_scripts/update_docs.py`), with the exception of the root index file  `index.rst`.
+
+### Declaring Optional Dependencies
+
+**Attention**: Make sure that any optional sensAI dependencies (which are not included in the `docs` tox environment) are added to `docs/conf.py` under `autodoc_mock_imports`. Otherwise the tox build will fail.
+
+### Notebooks
+
+`docs/index.rst` includes the names of notebooks which reside in the `notebooks/` folder. They are not initially present in the `docs/` folder, but any notebooks whose names are referenced in `index.rst` will be executed and saved with outputs to the `docs/` folder by a test in `notebooks/test_notebooks.py`.
+
+Therefore, in order for the docs build to work (without temporarily removing the notebook inclusions), it is necessary to run the aforementioned test at least once via
+
+    sh run_pytest_notebooks.sh
+
+For changes in notebooks to be reflected in the docs build, the test needs to be rerun.
+
+### Manually Running the Docs Build
 
-Make sure that any optional sensAI dependencies (which are not included in the `docs` tox environment) are added to `docs/conf.py` under `autodoc_mock_imports`.
+The docs build can be run without tox via 
+
+    sh build-docs.sh
+
+Results will be stored in `docs/build/`.
 
 # Creating a New Release
 
@@ -42,3 +86,53 @@ Make sure that any optional sensAI dependencies (which are not included in the `
      `bumpversion build --commit`
    * Continue with step 3.
 
+# Source-Level Directory Sync
+
+#### Details on the Synchonisation of a Source Directory within Your Project with the sensAI Repository
+
+We support the synchronisation of a branch in the sensAI repository with a directory within the git repository of your project which is to contain the sensAI source code (i.e. alternative #2 from above) via a convenient scripting solution.
+
+We consider two local repositories: the sensAI repository in directory `sensAI/` and your project in, for instance, directory `sensAI/../myprj/`. Let us assume that we want to synchronise branch `myprj-branch` in the sensAI repository with directory `myprj/src/sensai`.
+
+##### Synchronisation Script
+
+To perform the synchronisation, please create a script as follows, which you should save to `sensAI/sync.py`:
+
+```python
+import os
+from repo_dir_sync import LibRepo, OtherRepo
+
+r = LibRepo()
+r.add(OtherRepo("myprj", "myprj-branch", os.path.join("..", "myprj", "src", "sensai")))
+r.runMain()
+```
+
+You can add multiple other repositories if you so desire in the future.
+
+From directory `sensAI/` you can use the script in order to 
+
+* ***Push***: Update your project (i.e. `myprj/src/sensai`) with changes that were made in other projects by running `python sync.py myprj push`
+* ***Pull***: Update `myprj-branch` in the sensAI repository with changes made in your project by running `python sync.py myprj pull`
+
+##### Initialisation
+
+To initialise the synchronisation, proceed as follows:
+
+1. Create the branch `myprj-branch` in the sensAI repository, i.e. in `sensAI/` run this command:
+   `git branch myprj-branch master`
+2. Create the directory `myprj/src/sensai`.
+3. Make sure you have a `.gitignore` file in `myprj/` with at least the following entries:
+
+       *.pyc
+       __pycache__
+       *.bak
+       *.orig
+
+   Otherwise you may end up with unwanted tracked files after a synchronisation.
+4. Perform the initial *push*, i.e. in `sensAI/` run this command:
+   `python sync.py myprj push`
+
+##### Things to Keep in Mind
+
+* Both *push* and *pull* operations are always performed based on the branch that is currently checked out in `myprj/`. The best practice is to only use one branch for synchronisation, e.g. master.
+* *Push* and *pull* operations will make git commits in both repositories. Should an operation ever go wrong/not do what you intended, use `git reset --hard` to go back to the commits before the operation in both repositories.

README.md

@@ -24,7 +24,7 @@ In particular, sensAI provides ...
 
 ## Documentation
 
-Source code documentation and tutorials can be found [here](https://sensai.readthedocs.io/)
+Reference documentation and tutorials can be found [here](https://jambit.github.io/sensAI/docs/).
 
 ### Integrating sensAI into a Project
 
@@ -34,62 +34,14 @@ sensAI may be integrated into your project in several ways:
    Choose this option if you do not intend to make changes to sensAI in the context of your project.
 2. **Include sensAI's source code as a package within your project** (e.g. in `src/sensai`), which you synchronise with a sensAI branch.
    Choose this option if you intend to make changes to sensAI as you develop your project. When using this option, you (and others) may even make changes to sensAI in several branches of your project and even several projects using the same inclusion mechanism at the same time.
-   See below for details on how synchronisation works.
+   See developer documentation in README-dev.md for details on how synchronisation works.
 3. **Clone sensAI and add its source directory to your `PYTHONPATH`**.
    Choose this option if you potentially intend to make changes to sensAI but no one else working on your project will do the same and you will be modifying sensAI's source in no more than one branch at a time.
 
-#### Details on the Synchonisation of a Source Directory within Your Project with the sensAI Repository
-
-We support the synchronisation of a branch in the sensAI repository with a directory within the git repository of your project which is to contain the sensAI source code (i.e. alternative #2 from above) via a convenient scripting solution.
-
-We consider two local repositories: the sensAI repository in directory `sensAI/` and your project in, for instance, directory `sensAI/../myprj/`. Let us assume that we want to synchronise branch `myprj-branch` in the sensAI repository with directory `myprj/src/sensai`.
-
-##### Synchronisation Script
-
-To perform the synchronisation, please create a script as follows, which you should save to `sensAI/sync.py`:
-
-```python
-import os
-from repo_dir_sync import LibRepo, OtherRepo
-
-r = LibRepo()
-r.add(OtherRepo("myprj", "myprj-branch", os.path.join("..", "myprj", "src", "sensai")))
-r.runMain()
-```
-
-You can add multiple other repositories if you so desire in the future.
-
-From directory `sensAI/` you can use the script in order to 
-
-* ***Push***: Update your project (i.e. `myprj/src/sensai`) with changes that were made in other projects by running `python sync.py myprj push`
-* ***Pull***: Update `myprj-branch` in the sensAI repository with changes made in your project by running `python sync.py myprj pull`
-
-##### Initialisation
-
-To initialise the synchronisation, proceed as follows:
-
-1. Create the branch `myprj-branch` in the sensAI repository, i.e. in `sensAI/` run this command:
-   `git branch myprj-branch master`
-2. Create the directory `myprj/src/sensai`.
-3. Make sure you have a `.gitignore` file in `myprj/` with at least the following entries:
-
-       *.pyc
-       __pycache__
-       *.bak
-       *.orig
-
-   Otherwise you may end up with unwanted tracked files after a synchronisation.
-4. Perform the initial *push*, i.e. in `sensAI/` run this command:
-   `python sync.py myprj push`
-
-##### Things to Keep in Mind
-
-* Both *push* and *pull* operations are always performed based on the branch that is currently checked out in `myprj/`. The best practice is to only use one branch for synchronisation, e.g. master.
-* *Push* and *pull* operations will make git commits in both repositories. Should an operation ever go wrong/not do what you intended, use `git reset --hard` to go back to the commits before the operation in both repositories.
 
 ## Contributors
 
-sensAI is being developed by the artificial intelligence group at jambit GmbH.
+sensAI is being developed by the artificial intelligence group at [jambit GmbH](http://www.jambit.com) and by members of [appliedAI](www.appliedai.de).
 
 The main contributors are Dominik Jain, Michael Panchenko, Kristof Schröder and Magnus Winter.
 

build-docs.sh

@@ -1,3 +1,4 @@
+rm -rf docs/build
 python build_scripts/update_docs.py
 sphinx-build -W -b html docs docs/build
 

config.py

@@ -9,7 +9,7 @@
 
 __config_instance = None
 
-topLevelDirectory = os.path.dirname(__file__)
+topLevelDirectory = os.path.abspath(os.path.dirname(__file__))
 
 
 class __Configuration:
@@ -66,7 +66,7 @@ def _get_path(self, key: Union[str, List[str]], create=False) -> str:
         :return: the queried path
         """
         path_string = self._get_non_empty_entry(key)
-        path = os.path.abspath(path_string)
+        path = os.path.abspath(os.path.join(topLevelDirectory, path_string))
         if not os.path.exists(path):
             if isinstance(key, list):
                 key = ".".join(key)  # purely for logging

docs/conf.py

@@ -121,7 +121,8 @@ def findLineFromObjectName(sourceFile, objectName):
     "azure",
     "geopandas",
     "shapely",
-    "networkx"
+    "networkx",
+    "utm"
 ]
 
 # Render docu of __init__ methods
@@ -155,10 +156,14 @@ def findLineFromObjectName(sourceFile, objectName):
 # built documents.
 #
 # The full version, including alpha/beta/rc tags.
-release = pkg_resources.get_distribution(project).version
-# The short X.Y version.
-major_v, minor_v = release.split(".")[:2]
-version = f"{major_v}.{minor_v}"
+try:
+    version = pkg_resources.get_distribution(project).version
+    # The short X.Y version.
+    #major_v, minor_v = release.split(".")[:2]
+    #version = f"{major_v}.{minor_v}"
+except:
+    version = "dev"
+    log.warning(f"The {project} distribution was not found; using dummy version string '{version}'")
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/index.rst

@@ -5,8 +5,17 @@ sensAI - the Python library for sensible AI
    :caption: Guides and Tutorials
    :glob:
 
-   *
-
+   intro.ipynb
+   intro_old.ipynb
+   neural_networks.ipynb
+   tensor_models_pytorch_lightning.ipynb
+   coordinate_clustering.ipynb
+   clustering_evaluation.ipynb
+   tracking_experiments.ipynb
+
+..
+  Above, we can include any notebooks from ../notebooks; Within the build process, they will be copied
+  here by test_notebooks if they are referenced in this file.
 
 .. toctree::
    :caption: Modules

environment.yml

@@ -1,26 +1,46 @@
+# environment definition for development
+# NOTE: versions are mostly unpinned, because this facilitates conda dependency resolution. Also,
+# sensAI should be compatible to all (newer) versions of the dependencies. If it isn't, we need to specify 
+# an upper version bound in setup.py (where it matters the most) as well as here.
+# Compatibility with old (pinned) versions and the latest versions is tested in the tox build.
 name: sensai
 channels:
   - pytorch
-  - conda-forge
-  - anaconda
   - defaults
+  - conda-forge
 dependencies:
-  - python=3.7
-  - pytorch=1.4.0
-  - tensorflow=1.14.0
-  - pyqt=5.12
-  - psutil=5.6.7
-  - pytorch=1.4.0
-  - catboost=0.23
-  - xgboost=1.4.0
-  - pytest=5.4.1
-  - tox=3.14.6
-  - bump2version
-  - jupyter
-  - pyyaml=5.3
-  - lightgbm=3.1.1
-  - sphinx_rtd_theme
+  # basics
+  # we use Python 3.8 because it has better conda dependency support; the code must still be compatible to 3.7 (which is tested by the github build)
+  - python=3.8
   - pip
+  # optional sensai dependencies for development
+  # NOTE: catboost is disabled, because it causes conflicts
+  #- catboost
+  - pytorch
+  - pyqt
+  - psutil
+  - xgboost
+  - lightgbm
+  - utm=0.7.0
+  - geopandas
+  # for notebook support
+  - jupyter
+  # for tests
+  - pytest
+  # for docs build
+  - sphinx 
+  - sphinx_rtd_theme>=0.5.1
+  - nbsphinx
+  # for release process
+  - bump2version
   - pip:
-      - -r file:requirements.txt
-      - nbsphinx
+      # non-optional sensai dependencies
+      - -r requirements-relaxed.txt
+      # optional sensai pip dependencies
+      # clearml is only available through pip, and pulling tensorflow from conda causes dependency issues
+      - clearml
+      - tensorflow-cpu
+      - pytorch-lightning
+      # required for locally running the tox build (which will work on Linux only at this time)
+      - tox
+      - virtualenv