[Suggestion] Use .. plot:: directive to test code examples

[frankharkins]

Summary

This PR switches code examples to use Matplotlib's .. plot:: directive. This runs those code examples during documentation builds, failing the build if any code examples raise an exception.

Details

We replace code blocks with the plot directive, using the :nofigs: option. Any code inside the directive is run at build time. This doesn't change the appearance of the rendered docs.

.. plot::
   :include-source:
   :nofigs:

We can also use the :context: option to share state between code examples. This is helpful to avoid duplicating imports and for interspersing code examples with prose.

.. plot::
   :include-source:
   :nofigs:
   :context: reset

   from qiskit import QuantumCircuit

.. plot::
   :include-source:
   :nofigs:
   :context:

   qc = QuantumCircuit(1)

Advantages:

• No new tools: we already use .. plot:: in lots of places across the docs
• Flexibility: Used like this, the directive just checks the code runs, and won't fail if there are meaningless deviations in the output.

Potential problems:

• The term plot may be confusing to contributors since we're not plotting anything.
• It's a bit noisy repeating the same options (:include-source:, :nofig:) everywhere.

Impact on CI:

It's hard to tell since there's a lot of variation in the docs build step (20-30mins), but this doesn't seem to add noticeable time to CI (the two builds so far took 31 and 23 minutes).

---

Fixes Qiskit/documentation#1893

[coveralls]

Pull Request Test Coverage Report for Build 11571340391

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 17 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.02%) to 88.669%

---

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/lex.rs	5	92.73%
crates/qasm2/src/parse.rs	12	97.15%

Totals
Change from base Build 11526575270:	-0.02%
Covered Lines:	74890
Relevant Lines:	84460

---

💛 - Coveralls

[qiskit-bot]

One or more of the following people are relevant to this code:

• @enavarro51
• @Cryoris
• @Qiskit/terra-core
• @ajavadia
• @levbishop
• @mtreinish
• @nkanazawa1989
• @t-imamichi

[Eric-Arellano]

Great work! I like this approach. Sphinx is already such a learning curve that I think there's value in limiting the number of tools.

It's be useful to put the impact on CI's build from this change by looking at the Azure logs.

[jakelishman]

Thanks for doing all this, and thanks for the care you took in putting in clean-up code. That said, I'm still a bit nervous about merging any docs-build change that causes it to dirty the working directory while it's going, even if it's ephemeral - those stray files can easily get accidentally left behind, or the dev might cancel the docs build partway through, or so on.

Are the side-effect-y blocks something we can just leave as .. code-block::, or is there a neater way we might trick them into not actually writing to the filesystem?

[jakelishman]

There's a space remove here, but there's still leading indentation - is that what we expect, and is it displaying correctly?

[frankharkins]

Docs builds before and after render identically (the conversion script currently ignores the specified language, see Qiskit/documentation#2162). I think I removed the space to keep indentation a multiple of 2.

[jakelishman]

How come this one stayed as-is?

(Fwiw, python should be the default language for the Qiskit build, so I think it ought to have been highlighting correctly anyway.)

[frankharkins]

I think I wasn't sure how to make a pulse qobj and decided to leave it since this class is deprecated.

[jakelishman]

I'm not super wild about the docs build actually writing out dirty files into the working directory while we're going.

Could we either leave the above QPY stuff as .. code-block, or alternatively use a prior hidden context block to override open to be something that secretly returns io.BytesIO() objects (potentially wrapped to avoid their context managers clearing them up prematurely). Obviously that's a complete Python crime if it were real code, but if the context objects are localised and don't leak beyond this file, it'll keep the docs build entirely in-memory, not touching the file-system with temporaries.

[frankharkins]

Yeah that's a good point. The bytes object idea is a good one and I think it should work. Otherwise, I think it's fine not to test difficult cases like these.

EDIT: Switched to bytestream in 4370d52

[jakelishman]

Similar thing here - it feels better to me not to have the docs build actually write out to the filesystem as a side effect that we have to tidy up.

[frankharkins]

I agree, I think this one is simple enough that we don't need to test it.