From 527138bd9e87190f452ffb2c46572b343cb35ba9 Mon Sep 17 00:00:00 2001 From: Peter Karalekas Date: Tue, 15 Oct 2019 15:53:27 -0700 Subject: [PATCH] Start testing and reporting code coverage in the CI (#1052) --- .coveragerc | 5 ++- .gitlab-ci.yml | 29 +++++++------ CHANGELOG.md | 1 + CONTRIBUTING.md | 88 +++++++++++++++++++++++++++++----------- Dockerfile | 2 +- Makefile | 10 ++--- pyquil/_parser/README.md | 31 +++++++------- requirements.txt | 1 + 8 files changed, 107 insertions(+), 60 deletions(-) diff --git a/.coveragerc b/.coveragerc index 8da1c7ade..1411dc5be 100644 --- a/.coveragerc +++ b/.coveragerc @@ -1,2 +1,5 @@ [run] -omit = pyquil/tests/*,pyquil/_parser/gen2/*,pyquil/_parser/gen3/* +omit = + pyquil/_parser/gen3/* + pyquil/external/* + pyquil/tests/* diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e18f64477..84b5c00e5 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -19,41 +19,39 @@ test-py36: image: python:3.6 tags: - github - before_script: - - pip install tox script: - - make -e config - - tox -e py36 + - make install + - make requirements + - make test + coverage: '/TOTAL.*\s(\d+)%/' test-py37: image: python:3.7 tags: - github - before_script: - - pip install tox script: - - make -e config - - tox -e py37 + - make install + - make requirements + - make test style: image: python:3.6 tags: - github - before_script: - - pip install tox script: - - tox -e flake8 + - make install + - make requirements + - make style docs: image: python:3.6 tags: - github - before_script: - - pip install tox script: - apt-get update && apt-get install -y pandoc - - pandoc --from=markdown --to=rst --output=docs/source/changes.rst CHANGELOG.md - - tox -e docs + - make install + - make requirements + - make docs #################################################################################################### # MASTER-ONLY JOBS @@ -71,6 +69,7 @@ upload-testpypi: - make version > VERSION.txt - make install && make dist - make requirements && make upload + allow_failure: true #################################################################################################### # RELEASE-ONLY JOBS diff --git a/CHANGELOG.md b/CHANGELOG.md index 42a9cb34c..14732460e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ Changelog operations like creating and uploading a package (@karalekas, gh-1032). - As part of the CI, we now package and push to TestPyPI on every commit, which de-risks breaking the `setup.py` and aids with testing (@karalekas, gh-1017). +- We now calculate code coverage as part of the CI pipeline (@karalekas, gh-1052). ### Bugfixes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 07eb61352..b52dd56ab 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -93,10 +93,11 @@ Developer How-Tos ### Style Guidelines We use `flake8` to automatically lint the code and enforce style requirements as part of the -CI pipeline. You can run these style tests yourself locally by running `flake8 pyquil` in the -top-level directory of the repository. If you aren't presented with any errors, then that means -your code is good enough for the linter. In addition to these tests, we have a collection of -self-imposed style guidelines regarding typing, docstrings, and line length: +CI pipeline. You can run these style tests yourself locally by running `flake8 pyquil` (or +`make style`) in the top-level directory of the repository. If you aren't presented with any +errors, then that means your code is good enough for the linter. In addition to these tests, +we have a collection of self-imposed style guidelines regarding typing, docstrings, and line +length: 1. Use type hints for parameters and return types with the [PEP 484 syntax](https://www.python.org/dev/peps/pep-0484/). @@ -113,39 +114,73 @@ to run the tests, you should begin by spinning up these servers via `qvm -S` and respectively. Once this is done, run `pytest` in the top-level directory of pyQuil, and the full unit test suite will start! -**NOTE**: Some tests (particularly those related to operator estimation and readout -symmetrization) require a nontrivial amount of computation. For this reason, they have been marked -as slow and are not run by default unless `pytest` is given the `--runslow` option. For a -full, up-to-date list of these tests, you may invoke (from the top-level directory): +#### Slow Tests + +Some tests (particularly those related to operator estimation and readout symmetrization) +require a nontrivial amount of computation. For this reason, they have been marked +as slow and are not run by default unless `pytest` is given the `--runslow` option. +You will additionally need to provide the `pyquil` directory to the command, as `pytest` +doesn't try very hard to find the `conftest.py` file. The full command is as follows: + +```bash +pytest --runslow pyquil +``` + +For a full, up-to-date list of these slow tests, you may invoke (from the top-level directory): ```bash grep -A 1 -r pytest.mark.slow pyquil/tests/ ``` -**NOTE**: When making considerable changes to `operator_estimation.py`, we recommend that you set -the `pytest` option `--use-seed` to `False` to make sure you have not broken anything. You will -additionally need to provide the `pyquil` directory to the command, as `pytest` doesn't try -very hard to find the `conftest.py` file. The full command is as follows: +#### Seeded Tests + +When making considerable changes to `operator_estimation.py`, we recommend that you set the +`pytest` option `--use-seed` to `False` to make sure you have not broken anything. Because +this is another `conftest.py` option, you need to provide the directory again as above: ```bash pytest --use-seed=False pyquil ``` +#### Code Coverage + +In addition to testing the source code for correctness, we use `pytest` and the `pytest-cov` +plugin to calculate code coverage as part of the CI pipeline (via the `make test` command). +To produce this coverage report locally, run the following from the top-level directory: + +```bash +pytest --cov=pyquil +``` + +The coverage report omits the autogenerated parser code, the `external` module, and all of +the test code (as is specified in the [`.coveragerc`](.coveragerc) configuration file). + +#### Summary + +All of the above `pytest` variations can be mixed and matched according to what you're +trying to accomplish. For example, if you want to carefully test the operator estimation +code, run all of the slow tests, and also calculate code coverage, you could run: + +```bash +pytest --cov=pyquil --use-seed=False --runslow pyquil +``` + ### Building the Docs The [pyQuil docs](https://pyquil.readthedocs.io) build automatically as part of the CI pipeline. However, you can also build them locally to make sure that everything renders correctly. We use [Sphinx](http://www.sphinx-doc.org/en/master/) to build the documentation, and -then host it on [Read the Docs](https://readthedocs.org/) (RTD). Before you can build the docs -locally, you must make sure to install the additional Python-based requirements by -running `pip install -r requirements.txt`, which will pick up the Sphinx RTD theme and -autodocumentation functionality. In addition, you will need to install `pandoc` via your -favorite OS-level package manager (e.g. `brew`, `apt`, `yum`) in order to convert the -Changelog into reStructuredText (RST). Once you have done this, navigate into the `docs` -directory and run the following: +then host it on [Read the Docs](https://readthedocs.org/) (RTD). + +Before you can build the docs locally, you must make sure to install the additional +Python-based requirements by running `pip install -r requirements.txt`, which will pick up +the Sphinx RTD theme and autodocumentation functionality. In addition, you will need to +install `pandoc` via your favorite OS-level package manager (e.g. `brew`, `apt`, `yum`) in +order to convert the [Changelog](CHANGELOG.md) into reStructuredText (RST). Once you have done +this, run the following from the top-level directory: ```bash -make html +make docs ``` If the build is successful, then you can navigate to the newly-created `docs/build` @@ -183,9 +218,12 @@ docker run -it rigetti/forest:COMMIT_HASH Where `COMMIT_HASH` is replaced by the actual git commit hash. This will drop you into an `ipython` REPL with pyQuil installed and `quilc` / `qvm` servers running in the background. Exiting the REPL (via `C-d`) will additionally shut down the Docker container and return -you to the shell that ran the image. +you to the shell that ran the image. Docker images typically only have one running process, +but we leverage an [`entrypoint.sh`](entrypoint.sh) script to initialize the Forest SDK +runtime when the container starts up. -The image is defined by its [Dockerfile](Dockerfile), and it is additionally important to +The image is defined by its [Dockerfile](Dockerfile), along with a [`.dockerignore`](.dockerignore) +to indicate which files to omit when building the image. It is additionally important to note that this image depends on a collection of parent images, pinned to specific versions. This pinning ensures reproducibility, but requires that these versions be updated manually as necessary. The section of the Dockerfile that would need to be edited looks like this: @@ -248,7 +286,9 @@ following steps: After performing a release on GitHub, the next step is to build and push a new package to the Python Package Index (PyPI). This can be done locally in two steps (assuming you have the requisite credentials). First, run `make dist` from the top-level directory to -create a source distribution. Next, run the following: +create a source distribution. This will use the [`setup.py`](setup.py) to determine how +to produce the distribution, and will additionally include any files specified in the +[`MANIFEST.in`](MANIFEST.in). After the distribution is built, run the following: ```bash twine upload --repository pypi dist/* @@ -263,7 +303,7 @@ of the CI pipeline to ensure package robustness and enable easier integration te Every commit to `master` results in a new package published on pyQuil's Test PyPI project page [here](https://test.pypi.org/project/pyquil/). These packages have an additional number as part of their versioning scheme, which corresponds to the number of commits -the package is away from the latest tag (e.g. `v2.12.0.5` is 5 commits beyond `v2.12.0`), +the package is away from the latest tag (e.g. `v2.12.0.8` is 8 commits beyond `v2.12.0`), which can be determined via the command `git describe --tags`. If you wish to install a particular package from Test PyPI, run the following (changing the version as necessary): diff --git a/Dockerfile b/Dockerfile index 4b421803a..203cc9e7c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -30,5 +30,5 @@ WORKDIR /src/pyquil RUN pip install -e . # use an entrypoint script to add startup commands (qvm & quilc server spinup) -ENTRYPOINT ["./entrypoint.sh"] +ENTRYPOINT ["/src/pyquil/entrypoint.sh"] CMD ["ipython"] diff --git a/Makefile b/Makefile index 7daf49bf9..dfd8d3808 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ COMMIT_HASH=$(shell git rev-parse --short HEAD) +DEFAULT_QUILC_URL=tcp://localhost:5555 +DEFAULT_QVM_URL=http://localhost:5000 DOCKER_TAG=rigetti/forest:$(COMMIT_HASH) -QUILC_URL=tcp://localhost:5555 -QVM_URL=http://localhost:5000 .PHONY: all all: dist @@ -15,8 +15,8 @@ clean: .PHONY: config config: echo "[Rigetti Forest]" > ~/.forest_config - echo "qvm_address = ${QVM_URL}" >> ~/.forest_config - echo "quilc_address = ${QUILC_URL}" >> ~/.forest_config + echo "qvm_address = ${DEFAULT_QVM_URL}" >> ~/.forest_config + echo "quilc_address = ${DEFAULT_QUILC_URL}" >> ~/.forest_config cat ~/.forest_config .PHONY: dist @@ -51,7 +51,7 @@ style: .PHONY: test test: - pytest + pytest -v --runslow --cov=pyquil pyquil .PHONY: upload upload: diff --git a/pyquil/_parser/README.md b/pyquil/_parser/README.md index 9091c8253..94fe04c52 100644 --- a/pyquil/_parser/README.md +++ b/pyquil/_parser/README.md @@ -1,26 +1,29 @@ -# Quil Parser +The Quil Parser +=============== -## Introduction +Introduction +------------ -This package contains a number of items useful for parsing Quil programs, both with or without PyQuil. It uses the -ANTLR4 parser generator framework. +This package contains a number of items useful for parsing Quil programs, both with or without +pyQuil. It uses the ANTLR4 parser generator framework. -`Quil.g4` - This is the definitive reference grammar for Quil as described in the Quil paper. It is used to generate -the Python parser as well as parsers in other programming languages. +`Quil.g4` - This is the definitive reference grammar for Quil as described in the Quil paper. +It is used to generate the Python parser as well as parsers in other programming languages. -`PyQuilListener.py` - When the parser successfully consumes a Quil program it invokes various callbacks after -encountering different parts of the AST (gates, expressions, etc). This is the code that creates the PyQuil instructions. +`PyQuilListener.py` - When the parser successfully consumes a Quil program it invokes various +callbacks after encountering different parts of the AST (gates, expressions, etc). This is the +code that creates the pyQuil instructions. `gen3/` - Generated parser code for Python 3. Should be checked in but not hand modified. -## Running ANTLR +Running ANTLR +------------- -1. Install ANTLR4 and alias it to `antlr4` -2. cd to this directory +1. Install [ANTLR4](http://www.antlr.org/) and alias it to `antlr4` +2. Navigate to this directory (`pyquil/_parser`) 3. Generate the Python 3 parser code: `antlr4 -Dlanguage=Python3 -o gen3 Quil.g4` -## References +References +---------- Excellent ANTLR tutorial: https://tomassetti.me/antlr-mega-tutorial - -ANTLR download: http://www.antlr.org/ diff --git a/requirements.txt b/requirements.txt index 2b9afb0aa..517a53a0e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,6 +10,7 @@ rpcq >= 2.7.2 # test deps pytest +pytest-cov pytest-timeout pytest-rerunfailures requests-mock