Use MyST-NB to render jupyter notebook and markdown files

[weiji14]

Render the icepyx example jupyter notebooks directly at https://icepyx.readthedocs.io/en/development (i.e. without having to go to GitHub)!

Preview at https://icepyx--196.org.readthedocs.build/en/196/getting_started/examples.html

This is done using MyST-NB which parses jupyter notebooks (or markdown) in Sphinx. MyST-NB also acts as a replacement for:

• nbsphinx (for .ipynb files) added in fix some of the dead links in the docs and add new functions #28 at aa75a5a
• recommonmark (for .md files) added in Docs #49 at 1d79196

The powerful MyST (Markedly Structured Text) parser also allows much of the documentation to be written in Markdown instead of reStructuredText, which can make it easier for new folks wanting to make contributions to the documentation.

This PR also replaces numpydoc with Sphinx's built-in Napolean extension (they practically do the same thing). Edit: sticking with numpydoc for now as per #196 (comment)

P.S. Context for this Pull Request is to eventually revamp the front-facing docs page as per #110. This is really just a maintenance PR laying the groundwork by modernizing some of the Sphinx documentation dependencies, and fixing some issues along the way.

[codecov-io]

Codecov Report

Merging #196 (171dff9) into development (a582a27) will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           development     #196   +/-   ##
============================================
  Coverage        50.50%   50.50%           
============================================
  Files               17       17           
  Lines             1281     1281           
  Branches           280      280           
============================================
  Hits               647      647           
  Misses             581      581           
  Partials            53       53           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a582a27...171dff9. Read the comment docs.

[JessicaS11]

So far this looks like a great upgrade to our docs. Thanks @weiji14! I seem to remember, vaguely, some conversation around the pros/cons of using numpydoc instead of Napoleon, but I'll have to see if I can find any notes on that (@norlandrhagen, do you remember?).

My only major comment right now is that after a lot of back and forth we intentionally put the examples directory in the top level directory to make them easy to find. It's not intuitive for people to look in doc/source for them, even thought that's a "standard" home for them. I like the idea of them being rendered directly in the documentation, but also easy for people to find (since many find the repo before the docs). Thus, we should do some brainstorming around where they are available in the documentation (under "examples" or under "user guide"), where they reside in the repo, and what components of each notebook (if any besides the title) we want to show up in the left-hand menu bar, as well as how this ultimately relates to a user-contributed example gallery.

[weiji14]

My only major comment right now is that after a lot of back and forth we intentionally put the examples directory in the top level directory to make them easy to find. It's not intuitive for people to look in doc/source for them, even thought that's a "standard" home for them. I like the idea of them being rendered directly in the documentation, but also easy for people to find (since many find the repo before the docs).

Yes, I realize there was a lot of back and forth of moving the example notebooks, and I did try really hard to keep the *.ipynb files where they were in examples/ for discoverability. But it doesn't seem to work for MyST-NB, since the library only renders jupyter notebooks under the doc/ folder where sphinx is running. Maybe I'm missing some special configuration, but putting it in doc/user_guide was the easiest way I could get the notebook to show up in the docs.

Thus, we should do some brainstorming around where they are available in the documentation (under "examples" or under "user guide"), where they reside in the repo, and what components of each notebook (if any besides the title) we want to show up in the left-hand menu bar, as well as how this ultimately relates to a user-contributed example gallery.

Yes, an example gallery (as per #27) would be cool! I was catching up on the stalled issue at icesat2py/icepyx-gallery#1 the other day too. Maybe something to discuss for the meeting tomorrow (though I won't make it at 4am) or later this month.

[weiji14]

I'll try and put some work into this PR (and possibly #110) this week.

So far this looks like a great upgrade to our docs. Thanks @weiji14! I seem to remember, vaguely, some conversation around the pros/cons of using numpydoc instead of Napoleon, but I'll have to see if I can find any notes on that (@norlandrhagen, do you remember?).

There's a stackoverflow question on it at https://stackoverflow.com/questions/36715801/difference-between-sphinxcontrib-napoleon-and-numpy-numpydoc#45009645. Just to quote it:

default behavior of napoleon links to known datatypes in the python documentation, and it is slightly more condensed (numpydoc displays a bit like how it appears in the docstring)

Functionally, they seem to do about the same thing (save us from writing in raw Sphinx syntax). I can revert the Napolean addition if you prefer to keep using NumpyDoc for icepyx?

My only major comment right now is that after a lot of back and forth we intentionally put the examples directory in the top level directory to make them easy to find. It's not intuitive for people to look in doc/source for them, even thought that's a "standard" home for them. I like the idea of them being rendered directly in the documentation, but also easy for people to find (since many find the repo before the docs). Thus, we should do some brainstorming around where they are available in the documentation (under "examples" or under "user guide"), where they reside in the repo, and what components of each notebook (if any besides the title) we want to show up in the left-hand menu bar, as well as how this ultimately relates to a user-contributed example gallery.

Another idea would be to keep the examples/ folder for visibility, but put a link in the examples/README.md to point to doc/source/example_notebooks/, similar to how geopandas does it at https://github.com/geopandas/geopandas/tree/master/examples.

[codecov-commenter]

Codecov Report

Merging #196 (2cbf85b) into development (be9427f) will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           development     #196   +/-   ##
============================================
  Coverage        51.91%   51.91%           
============================================
  Files               26       26           
  Lines             1855     1855           
  Branches           396      396           
============================================
  Hits               963      963           
  Misses             829      829           
  Partials            63       63           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be9427f...2cbf85b. Read the comment docs.

[JessicaS11]

Finally getting into this PR again after way too long a hiatus. I rebased the branch onto the most recent development to avoid conflicts (hence the "new" commits). Thanks for your patience, @weiji14!

Functionally, they seem to do about the same thing (save us from writing in raw Sphinx syntax). I can revert the Napolean addition if you prefer to keep using NumpyDoc for icepyx?

I personally like the look of Numpydoc over Napolean. From your notes and what I've read, it comes down mostly to personal preference which one we use... can we stick wiht Numpydoc for now?

[JessicaS11]

Another idea would be to keep the examples/ folder for visibility, but put a link in the examples/README.md to point to doc/source/example_notebooks/, similar to how geopandas does it at https://github.com/geopandas/geopandas/tree/master/examples.

It's really weird to me that you can't use links to get to files outside of docs. It looks like it could be in the works, as this feature has been implemented for Markdown-style images and links. I played around with using {include} as in the example, but then it was just printing the command on the page rather than executing it. Next week I'll work on moving the rest of the examples and putting them all under a single examples tab with good references as you suggested r.e. geopandas.

[fperez]

I wanted to ping @choldgraf on this one :) Chris, do you have any thoughts on the examples management issue discussed above by @weiji14 and @JessicaS11? It seems that having flexibility on their location, for the purposes of regular use of the repo, is a reasonable use pattern, but it seems to clash with the rest of the build process a bit...

A rather brutish hack would be to add a recursive copy operation against he examples dir at the start of the build (we could do it with hardlinks to go fast - os.link from the stdlib works on Unix and Windows, so a quick recursive copy can be made with a single shutil.copytree(src, dst, copy_function=os.link) call.

But that feels crude, so perhaps @choldgraf has better ideas ;)

[JessicaS11]

Just wanted to add a general note here: with the notebooks now being rendered, they create a pretty long, confusing list that makes it hard to see what the actual example are. The examples have been updated and rearranged in #222, including some work to standardize headings, so we can make any additional changes to the examples themselves there and limit this PR to the doc generation itself.

[weiji14]

Right, I'm slowly trying to catch up on what's happening here, too many things happening at once! Maybe we should have a separate, dedicate clean-up/reorganization PR to do all the renaming/file moving? Or do you prefer to just finalize #222 first before merging this #196 PR?

[JessicaS11]

I like the idea of finalizing #222 and then merging this PR#196. To reduce git friction, I'll go ahead and make sure I don't change the example file names in #222 and rename them here instead.

[JessicaS11]

Filenames in #222 don't change, just add a few new notebooks. I'll let them land with the rest of the notebooks there, and then we'll have to move them on this branch before we merge #196.

[choldgraf]

Hmmm one thing you could try is saving your notebooks as MyST markdown notebooks and then using an include directive in your docs, but that may not be ideal if you want them as ipynb.

To be honest i think the most reasonable solution is to indeed add a copytree in the conf.py so that it gets moved over at build time, and then add that folder to your gitignore so it isn't in the repository more than once. But annoyingly Sphinx has no way to reference file outside the documentation root from the toctree

[JessicaS11]

Thanks @choldgraf for that feedback! I'll solicit some input from a few users, but I suspect that we can get away with the GeoPandas approach of putting some links and notes in the top level examples directory and then just house the notebooks themselves within the docs directory (which seems to be standard practice and I think made it a lot harder for scientists to find them before there was also a docs page where they were rendered!).

[fperez]

@JessicaS11 in case it's helpful, here's my (very old) copy_trees.py script that I've used for ages for this task in sphinx builds...

[JessicaS11]

@weiji14 When you get a chance (no rush - January is fine!) can you look this PR over again since I was the last one to make a bunch of edits? Then I can give it an approving review and we can merge it. Note that I created Issue #257 to improve the notebook organization after this initial PR to get the functionality there.

[weiji14]

@weiji14 When you get a chance (no rush - January is fine!) can you look this PR over again since I was the last one to make a bunch of edits? Then I can give it an approving review and we can merge it. Note that I created Issue #257 to improve the notebook organization after this initial PR to get the functionality there.

Just from a quick scan, I do feel like there's too many nested headings. Do you think it's possible to remove the 'Example notebooks' level, and just have everything under 'Examples'?

I'll try and give this a closer look next week after AGU is over. Too many things on my plate right now...

[JessicaS11]

Do you think it's possible to remove the 'Example notebooks' level, and just have everything under 'Examples'?

Done - great catch.

I'll try and give this a closer look next week after AGU is over. Too many things on my plate right now...

No rush - I just wanted to document it was ready for the next step.

[weiji14]

Hi @JessicaS11, I've made a few more minor updates to fix some Sphinx doc build warnings. Also, I've moved the notebook examples into a dedicated section on the sidebar to improve visibility:

Let me know if you're ok with this, otherwise I can revert commit a161b42 to have the examples be nested under Getting Started -> Examples as before:

[JessicaS11]

Also, I've moved the notebook examples into a dedicated section on the sidebar to improve visibility:

I really like this change, but I'm concerned that it will make the menu bar long and diluted as we add new examples. To give us time to come up with an plan (perhaps we can have a few main categories of example type that get listed on the menu bar, with individual examples listed within?), I reverted the change in this PR and after we merge will create a new branch with it that we can keep working on. That will also have your commit to increase the heading levels, since we need to do a go-through of the examples to standardize heading levels anyway.