Documentation page outline

[rwegener2]

What has been built?

This PR provides a potential outline for a central documentation page for Pangeo Forge, as discussed in Issue #25.

Most parts of this page haven't been developed yet so we probably don't want to actually merge all of these suggestions, but I hope that what is in this PR could provide an outline for what we want the centralized docs page to become. As always I'm very open to feedback.

How was it done?

@cisaacstern has already typed out the steps:

1. Add the page itself as a file called Documention.vue to src/components/
2. Add a reference to it in src/router/index.js
3. Change the Documentation href in src/App.vue

Note that because the link in the "Documentation" button is hardcoded you have to manually navigate to localhost:8080/docs to see the new page.

[cisaacstern]

@rwegener2, I've added comments inline, which hopefully should get us closer to a deployable version of this new page. Very curious to hear your thoughts.

[cisaacstern]

Should this link to something? If so, what/where? Perhaps a new page at https://pangeo-forge.org/docs/basics or similar? Or can this question be answered without it's own page?

[rwegener2]

I am open to hearing other opinions but at first pass I was thinking it could just point to https://github.com/pangeo-forge/roadmap. The readme information specifically on the Inspiration, Vision, and one paragraph per technical concept I think is an appropriate level of information.

[rwegener2]

In the latest commit I pointed this to https://pangeo-forge.readthedocs.io/en/latest/what_is_pangeo_forge.html. The content that is there I think is nice. I also imagine a technical description similar to this could be a nice thing to add there at some point, but I know there are a lot of potential avenues to pursue.

[cisaacstern]

Should this link to https://pangeo-forge.readthedocs.io/en/latest/tutorials/index.html ?

I haven't reviewed the content of those tutorials recently and don't know how representative/accessible they are.

[rwegener2]

Those tutorials are nice, but I think the goal of an introduction tutorial is specific - an intro tutorial should walk a brand new person step by step through the simplest possible meaningful version of deploying a data transformation. I see that as a new tutorial and I'm not sure right now if I think it should live as a pangeo-forge-recipes tutorial or a Pangeo Forge-wide tutorial. Conceptually I want it to be the latter, but I'm not sure if there is enough content for that. I discuss more in my comment below.

[rwegener2]

Updated to a link to the Introduction Tutorial section of the readthedocs page that was created in the most recent PR.

[cisaacstern]

What do you think the Introduction Tutorial should contain? We already have some Jupyter Notebooks rendered to HTML, i.e. the tutorials at https://pangeo-forge.readthedocs.io/en/latest/tutorials/index.html ...

..but my inclination is that we should be steering people away from thinking of notebooks as the way to use pangeo-forge-recipes, and more towards developing Python modules, i.e. recipe.py files.

So perhaps this should link to a new page called ... https://pangeo-forge.org/docs/intro-tutorial ?

[rwegener2]

I don't know the code well enough to have a notebooks vs. scripts opinion but I trust your intuition, @cisaacstern.

As for content I talk more about that in my comment below, but I think the introduction tutorial should walk someone through deploying a data transformation from start to finish. The goal is for someone to walk away feeling like "I can deploy data transformations with Pangeo Forge". A simple but meaningful data goal would be great (maybe geotiff to cog with simple or accessible input/output locations?)

[cisaacstern]

Do you have a link of a Guide for another open source package that you really like? I'm having difficulty picturing what the Guide looks like.

[rabernat]

I think the existing documentation for pangeo_forge_recipes can evolve into a user guide. The Xarray User Guide is a good inspiration.

[rwegener2]

I like the xarray User Guide pretty well. When I'm saying Guides I'm thinking of the "HowTo" (goal oriented: "take the reader through the steps required to solve a real-world problem") and "Explanation" (understanding oriented: "clarify and illuminate a particular topic. They broaden the documentation’s coverage of a topic.") sections of the Divio documentation construct.
While that was my original thought I think we can think more fluidly than that if we want. xarray sort of does them together and I think those docs are nice.

[cisaacstern]

Suggested change

	deploying your first data transformation pipeline.</li>
	deploying your first data transformation recipe.</li>

My feeling is we should avoid "pipeline" in favor of "recipe", unless we are referring to the specific case of execution pipelines, e.g. in base.py here. But most users of Pangeo Forge will not actually touch that part of the code.

[rwegener2]

I have no attachment to "pipeline" so that part is fine. I am hesitant to use the word "recipe" except when we really mean specifically a recipe. My leaning is that for the intro tutorial we want to encapsulate more than just walking a user through recipe creation, we want to go all the way from idea to new transformed data. @cisaacstern What do you think of just removing the final word and writing deploying your first data transformation?

[rwegener2]

I just dropped "pipeline" altogether, but we could of course change to "recipe" if that is more appropriate

[rabernat]

Fantastic! It will be great to have this documentation jumping-off point on this main website

I propose we put as much of the actual documentation content as possible in https://github.com/pangeo-forge/pangeo-forge-recipes/tree/master/docs. This will require reorganizing things a bit. Much of the current documentation will move down a level in terms of nesting, to make way for higher level content (getting started, high-level tutorial, etc.) How does that sound?

[rabernat]

Over in pangeo-forge/pangeo-forge-recipes#178, I have expanded the base documentation page to allow for the new content.

My thinking is that we have this html page essentially mirror the main index.md page of https://pangeo-forge.readthedocs.io/. All the documentation links will point to pangeo-forge.readthedocs.io. It will be easier to write long, narrative documentation inside https://github.com/pangeo-forge/pangeo-forge-recipes/tree/master/docs because

• We can write markdown instead of html
• We can leverage sphinx features (xrefs, TOCs, etc)

We could also easily create a DNS alias to point docs.pangeo-forge.io to pangeo-forge.readthedocs.io if people think that would help make things seem more coherent.

[rwegener2]

Thanks so much for the comments and review @cisaacstern and @rabernat! I'm a little tied up this week but I'll dig into this and provide some more thoughts on organizing the pangeo-forge-recipes docs pages over the weekend.

[rwegener2]

This is all really great feedback. I hope I've mostly answered questions above. I did make one new change in response to comments which was to rewrite the "How the documentation is organized" section to emphasize that there are multiple repositories.

Links and Hierarchy

There were several comments about links and an ongoing discussion from the meeting about the hierarchy of the https://pangeo-forge.readthedocs.io site, so I attached a figure I drew while I was thinking it through. This might be too sloppy to be helpful so if that is the case I apologize and I can for sure upload a clearer visual.

Grey arrows show where links go to. Headers on the left follow the current docs page in this PR. Headers on the right are link locations and proposed organization for the pangeo-forge-recipes docs.

If I am able to figure out how to get the readthedocs page running locally I'm happy to put some of this structure into code for a clearer example.

Intro Tutorial

There were a few comments about the the introduction tutorial. I think the goal of the intro tutorial should be to walk a brand new person step by step through the simplest possible version of deploying a meaningful data transformation. This feels like an important piece of the docs to me because anyone who we loose on the intro tutorial we are probably loosing off the ecosystem. Creating a sense of "I can do this" is an important outcome. I loved how the Divio documentation system thinks about tutorials. Tutorial goals are summarized in a few of their section headers: "Get the user started" -- "Ensure the user sees results immediately" -- "Focus on concrete steps, not abstract concepts" -- "Provide the minimum necessary explanation".

In the context of Pangeo Forge my understanding is that running a data transformation has multiple steps:

• fork the pangeo-forge-recipes library (maybe?)
• choose a bakery from the recipe box on which to store your data and run the transformation
• follow the docs to use Python to create a recipe file and a yaml config file
• fork the staged-recipes repo, push your recipe and make a PR. once the PR is merged this will automatically create a feedstock
• hit "go" on the feedstock
• wait patiently and excitedly while your data processes
• go find your data in the bakery

These might not actually be correct but my perception is that that while writing the recipe is for sure the biggest step, it isn't the only one. If it is technically possible I think our intro tutorial should span from the first to the last bullet.

[rabernat]

Rachel, thanks for this super useful drawing and high-level thinking.

Let me propose a path forward. Rather than solving all our documentation problems in a single, massive PR, let's try to move incrementally. Here is a possible order of operations.

• For now, we leave aside the formal documentation of bakeries, as it hasn't really be written yet. We can just point to the GitHub repos, which contain verbose READMEs.
• We finish reviewing Restructure docs pangeo-forge-recipes#178 and align on the revised structure of that documentation page. We agree that, for now, https://pangeo-forge.readthedocs.io/ will have all of the 🍗 of the documentation (tutorials, guides, API docs, etc.). We have a placeholder page for the intro tutorial.
• Once that is merged, we update this PR ("central docs page") with the correct links to https://pangeo-forge.readthedocs.io/ and merge it.

After that, going forward, we can have future, standalone PRs to address

• The intro tutorial
• "What is Pangeo Forge"
• The developer guide
• The bakery deployment process
• etc etc

How does this sound to everyone?

[rwegener2]

I like that a lot @rabernat, thanks for typing that out in such a clear and organized way.

Once the Restructure docs pangeo-forge-recipes#178 gets merged I'm happy to go through and update the links in this PR.

[rabernat]

pangeo-forge/pangeo-forge-recipes#178 has been merged. Now we need to update this PR.

@rwegener2 - if you don't have time for this (I know you're not officially working on this project any more!) just let us know. Some one else would be happy to take it over. But either way, we should get it done so we can move forward.

[rwegener2]

I'm sorry I've been so unresponsive @rabernat.

I just updated the PR with links corresponding to the readthedocs page. I also updated some of the text in "How the Documentation is Organized" in line with what is written on the readthedocs homepage to explain the contents of the repo.

I'll keep an eye out for any follow up comments so we can get this merged asap.

[rwegener2]

Just wanted to circle back on this PR. My impression after the last meeting was that this is ready to merge. @rabernat or @cisaacstern let me know if you are waiting on me for anything or if I can help.

[rabernat]

Yes!

[rabernat]

Going forward, my one concern is that we now have to update two places with essentially the same information

• https://github.com/pangeo-forge/pangeo-forge-vue-website/blob/main/src/components/Documentation.vue
• https://github.com/pangeo-forge/pangeo-forge-recipes/blob/master/docs/index.md

Would it perhaps not be simpler to have the navbar "docs" link from pangeo-forge.org point directly at https://pangeo-forge.readthedocs.io/?

[cisaacstern]

Would it perhaps not be simpler to have the navbar "docs" link from pangeo-forge.org point directly at https://pangeo-forge.readthedocs.io/?

I'm inclined towards this, but @rwegener2 has consistently pointed out dimensions that I hadn't considered, so open to being persuaded otherwise, if you have a different view, Rachel.