Start testing and reporting code coverage in the CI (#1052)

.coveragerc

@@ -1,2 +1,5 @@
 [run]
-omit = pyquil/tests/*,pyquil/_parser/gen2/*,pyquil/_parser/gen3/*
+omit =
+    pyquil/_parser/gen3/*
+    pyquil/external/*
+    pyquil/tests/*

.gitlab-ci.yml

@@ -19,41 +19,39 @@ test-py36:
   image: python:3.6
   tags:
     - github
-  before_script:
-    - pip install tox
   script:
-    - make -e config
-    - tox -e py36
+    - make install
+    - make requirements
+    - make test
+  coverage: '/TOTAL.*\s(\d+)%/'
 
 test-py37:
   image: python:3.7
   tags:
     - github
-  before_script:
-    - pip install tox
   script:
-    - make -e config
-    - tox -e py37
+    - make install
+    - make requirements
+    - make test
 
 style:
   image: python:3.6
   tags:
     - github
-  before_script:
-    - pip install tox
   script:
-    - tox -e flake8
+    - make install
+    - make requirements
+    - make style
 
 docs:
   image: python:3.6
   tags:
     - github
-  before_script:
-    - pip install tox
   script:
     - apt-get update && apt-get install -y pandoc
-    - pandoc --from=markdown --to=rst --output=docs/source/changes.rst CHANGELOG.md
-    - tox -e docs
+    - make install
+    - make requirements
+    - make docs
 
 ####################################################################################################
 # MASTER-ONLY JOBS
@@ -71,6 +69,7 @@ upload-testpypi:
     - make version > VERSION.txt
     - make install && make dist
     - make requirements && make upload
+  allow_failure: true
 
 ####################################################################################################
 # RELEASE-ONLY JOBS

CHANGELOG.md

@@ -20,6 +20,7 @@ Changelog
     operations like creating and uploading a package (@karalekas, gh-1032).
 -   As part of the CI, we now package and push to TestPyPI on every commit, which
     de-risks breaking the `setup.py` and aids with testing (@karalekas, gh-1017).
+-   We now calculate code coverage as part of the CI pipeline (@karalekas, gh-1052).
 
 ### Bugfixes
 

CONTRIBUTING.md

@@ -93,10 +93,11 @@ Developer How-Tos
 ### Style Guidelines
 
 We use `flake8` to automatically lint the code and enforce style requirements as part of the
-CI pipeline. You can run these style tests yourself locally by running `flake8 pyquil` in the
-top-level directory of the repository. If you aren't presented with any errors, then that means
-your code is good enough for the linter. In addition to these tests, we have a collection of
-self-imposed style guidelines regarding typing, docstrings, and line length:
+CI pipeline. You can run these style tests yourself locally by running `flake8 pyquil` (or
+`make style`) in the top-level directory of the repository. If you aren't presented with any
+errors, then that means your code is good enough for the linter. In addition to these tests,
+we have a collection of self-imposed style guidelines regarding typing, docstrings, and line
+length:
 
 1. Use type hints for parameters and return types with the
    [PEP 484 syntax](https://www.python.org/dev/peps/pep-0484/).
@@ -113,39 +114,73 @@ to run the tests, you should begin by spinning up these servers via `qvm -S` and
 respectively. Once this is done, run `pytest` in the top-level directory of pyQuil, and the
 full unit test suite will start!
 
-**NOTE**: Some tests (particularly those related to operator estimation and readout
-symmetrization) require a nontrivial amount of computation. For this reason, they have been marked 
-as slow and are not run by default unless `pytest` is given the `--runslow` option. For a 
-full, up-to-date list of these tests, you may invoke (from the top-level directory):
+#### Slow Tests
+
+Some tests (particularly those related to operator estimation and readout symmetrization)
+require a nontrivial amount of computation. For this reason, they have been marked
+as slow and are not run by default unless `pytest` is given the `--runslow` option.
+You will additionally need to provide the `pyquil` directory to the command, as `pytest`
+doesn't try very hard to find the `conftest.py` file. The full command is as follows:
+
+```bash
+pytest --runslow pyquil
+```
+
+For a full, up-to-date list of these slow tests, you may invoke (from the top-level directory):
 
 ```bash
 grep -A 1 -r pytest.mark.slow  pyquil/tests/
 ```
 
-**NOTE**: When making considerable changes to `operator_estimation.py`, we recommend that you set
-the `pytest` option `--use-seed` to `False` to make sure you have not broken anything. You will
-additionally need to provide the `pyquil` directory to the command, as `pytest` doesn't try
-very hard to find the `conftest.py` file. The full command is as follows:
+#### Seeded Tests
+
+When making considerable changes to `operator_estimation.py`, we recommend that you set the
+`pytest` option `--use-seed` to `False` to make sure you have not broken anything. Because
+this is another `conftest.py` option, you need to provide the directory again as above:
 
 ```bash
 pytest --use-seed=False pyquil
 ```
 
+#### Code Coverage
+
+In addition to testing the source code for correctness, we use `pytest` and the `pytest-cov`
+plugin to calculate code coverage as part of the CI pipeline (via the `make test` command).
+To produce this coverage report locally, run the following from the top-level directory:
+
+```bash
+pytest --cov=pyquil
+```
+
+The coverage report omits the autogenerated parser code, the `external` module, and all of
+the test code (as is specified in the [`.coveragerc`](.coveragerc) configuration file).
+
+#### Summary
+
+All of the above `pytest` variations can be mixed and matched according to what you're
+trying to accomplish. For example, if you want to carefully test the operator estimation
+code, run all of the slow tests, and also calculate code coverage, you could run:
+
+```bash
+pytest --cov=pyquil --use-seed=False --runslow pyquil
+```
+
 ### Building the Docs
 
 The [pyQuil docs](https://pyquil.readthedocs.io) build automatically as part of the CI pipeline.
 However, you can also build them locally to make sure that everything renders correctly. We use
 [Sphinx](http://www.sphinx-doc.org/en/master/) to build the documentation, and
-then host it on [Read the Docs](https://readthedocs.org/) (RTD). Before you can build the docs
-locally, you must make sure to install the additional Python-based requirements by
-running `pip install -r requirements.txt`, which will pick up the Sphinx RTD theme and
-autodocumentation functionality. In addition, you will need to install `pandoc` via your
-favorite OS-level package manager (e.g. `brew`, `apt`, `yum`) in order to convert the
-Changelog into reStructuredText (RST). Once you have done this, navigate into the `docs`
-directory and run the following:
+then host it on [Read the Docs](https://readthedocs.org/) (RTD).
+
+Before you can build the docs locally, you must make sure to install the additional
+Python-based requirements by running `pip install -r requirements.txt`, which will pick up
+the Sphinx RTD theme and autodocumentation functionality. In addition, you will need to
+install `pandoc` via your favorite OS-level package manager (e.g. `brew`, `apt`, `yum`) in
+order to convert the [Changelog](CHANGELOG.md) into reStructuredText (RST). Once you have done
+this, run the following from the top-level directory:
 
 ```bash
-make html
+make docs
 ```
 
 If the build is successful, then you can navigate to the newly-created `docs/build`
@@ -183,9 +218,12 @@ docker run -it rigetti/forest:COMMIT_HASH
 Where `COMMIT_HASH` is replaced by the actual git commit hash. This will drop you into an
 `ipython` REPL with pyQuil installed and `quilc` / `qvm` servers running in the background.
 Exiting the REPL (via `C-d`) will additionally shut down the Docker container and return
-you to the shell that ran the image.
+you to the shell that ran the image. Docker images typically only have one running process,
+but we leverage an [`entrypoint.sh`](entrypoint.sh) script to initialize the Forest SDK
+runtime when the container starts up.
 
-The image is defined by its [Dockerfile](Dockerfile), and it is additionally important to
+The image is defined by its [Dockerfile](Dockerfile), along with a [`.dockerignore`](.dockerignore)
+to indicate which files to omit when building the image. It is additionally important to
 note that this image depends on a collection of parent images, pinned to specific versions.
 This pinning ensures reproducibility, but requires that these versions be updated manually
 as necessary. The section of the Dockerfile that would need to be edited looks like this:
@@ -248,7 +286,9 @@ following steps:
 After performing a release on GitHub, the next step is to build and push a new package
 to the Python Package Index (PyPI). This can be done locally in two steps (assuming you
 have the requisite credentials). First, run `make dist` from the top-level directory to
-create a source distribution. Next, run the following:
+create a source distribution. This will use the [`setup.py`](setup.py) to determine how
+to produce the distribution, and will additionally include any files specified in the
+[`MANIFEST.in`](MANIFEST.in). After the distribution is built, run the following:
 
 ```bash
 twine upload --repository pypi dist/*
@@ -263,7 +303,7 @@ of the CI pipeline to ensure package robustness and enable easier integration te
 Every commit to `master` results in a new package published on pyQuil's Test PyPI project
 page [here](https://test.pypi.org/project/pyquil/). These packages have an additional
 number as part of their versioning scheme, which corresponds to the number of commits
-the package is away from the latest tag (e.g. `v2.12.0.5` is 5 commits beyond `v2.12.0`),
+the package is away from the latest tag (e.g. `v2.12.0.8` is 8 commits beyond `v2.12.0`),
 which can be determined via the command `git describe --tags`. If you wish to install a
 particular package from Test PyPI, run the following (changing the version as necessary):
 

Dockerfile

@@ -30,5 +30,5 @@ WORKDIR /src/pyquil
 RUN pip install -e .
 
 # use an entrypoint script to add startup commands (qvm & quilc server spinup)
-ENTRYPOINT ["./entrypoint.sh"]
+ENTRYPOINT ["/src/pyquil/entrypoint.sh"]
 CMD ["ipython"]

Makefile

@@ -1,7 +1,7 @@
 COMMIT_HASH=$(shell git rev-parse --short HEAD)
+DEFAULT_QUILC_URL=tcp://localhost:5555
+DEFAULT_QVM_URL=http://localhost:5000
 DOCKER_TAG=rigetti/forest:$(COMMIT_HASH)
-QUILC_URL=tcp://localhost:5555
-QVM_URL=http://localhost:5000
 
 .PHONY: all
 all: dist
@@ -15,8 +15,8 @@ clean:
 .PHONY: config
 config:
 	echo "[Rigetti Forest]" > ~/.forest_config
-	echo "qvm_address = ${QVM_URL}" >> ~/.forest_config
-	echo "quilc_address = ${QUILC_URL}" >> ~/.forest_config
+	echo "qvm_address = ${DEFAULT_QVM_URL}" >> ~/.forest_config
+	echo "quilc_address = ${DEFAULT_QUILC_URL}" >> ~/.forest_config
 	cat ~/.forest_config
 
 .PHONY: dist
@@ -51,7 +51,7 @@ style:
 
 .PHONY: test
 test:
-	pytest
+	pytest -v --runslow --cov=pyquil pyquil
 
 .PHONY: upload
 upload:

pyquil/_parser/README.md

@@ -1,26 +1,29 @@
-# Quil Parser
+The Quil Parser
+===============
 
-## Introduction
+Introduction
+------------
 
-This package contains a number of items useful for parsing Quil programs, both with or without PyQuil. It uses the 
-ANTLR4 parser generator framework.
+This package contains a number of items useful for parsing Quil programs, both with or without
+pyQuil. It uses the ANTLR4 parser generator framework.
 
-`Quil.g4` - This is the definitive reference grammar for Quil as described in the Quil paper. It is used to generate
-the Python parser as well as parsers in other programming languages.
+`Quil.g4` - This is the definitive reference grammar for Quil as described in the Quil paper.
+It is used to generate the Python parser as well as parsers in other programming languages.
 
-`PyQuilListener.py` - When the parser successfully consumes a Quil program it invokes various callbacks after
-encountering different parts of the AST (gates, expressions, etc). This is the code that creates the PyQuil instructions.
+`PyQuilListener.py` - When the parser successfully consumes a Quil program it invokes various
+callbacks after encountering different parts of the AST (gates, expressions, etc). This is the
+code that creates the pyQuil instructions.
 
 `gen3/` - Generated parser code for Python 3. Should be checked in but not hand modified.
 
-## Running ANTLR
+Running ANTLR
+-------------
 
-1. Install ANTLR4 and alias it to `antlr4`
-2. cd to this directory
+1. Install [ANTLR4](http://www.antlr.org/) and alias it to `antlr4`
+2. Navigate to this directory (`pyquil/_parser`)
 3. Generate the Python 3 parser code: `antlr4 -Dlanguage=Python3 -o gen3 Quil.g4` 
 
-## References
+References
+----------
 
 Excellent ANTLR tutorial: https://tomassetti.me/antlr-mega-tutorial
-
-ANTLR download: http://www.antlr.org/

requirements.txt

@@ -10,6 +10,7 @@ rpcq >= 2.7.2
 
 # test deps
 pytest
+pytest-cov
 pytest-timeout
 pytest-rerunfailures
 requests-mock