Parent task: Set up subprojects for Kedro-Viz and kedro-datasets docs

[stichbury]

We are thinking about using a subproject for Viz docs to enable them to be versioned according to the Viz release and updated along with those releases rather than linked to Framework release schedules.

The team plan to make multiple releases for collaborative experiment tracking and would like to manage the docs for different versions autonomously. We'd publish to https://docs.kedro.org/kedro-viz

However, does RTD support subproject is in a different repo? I've only seen mention of subprojects being in a single repo? As far as we can tell, this should be fine, and should be a useful model for other docs like those for datasets and other plugins. We need to run a pilot to see if it's feasible to do this, and how long it'll take.

[astrojuanlu]

Indeed RTD subprojects can be a separate repository. A subproject has to be a proper RTD project to begin with, so we'd need to set up a RTD project for Kedro Viz first, and then add it to the Kedro main project:

https://docs.readthedocs.io/en/stable/guides/subprojects.html

[tynandebold]

How much of a lift is it to move the Viz docs out of the Framework repo and into it's own? The process doesn't seem too hard, since we already have all the parts.

Or would it make more sense to have the Viz docs themselves in the Viz repo and use that as the subproject?

[astrojuanlu]

Or would it make more sense to have the Viz docs themselves in the Viz repo and use that as the subproject?

This is what we have in mind. The process would be like this:

1. Create a Sphinx project in kedro-viz
2. Create a RTD project for kedro-viz
3. Use that RTD project as subproject of kedro
4. Profit

It should work.

[tynandebold]

Great. Let me know if I can help.

[astrojuanlu]

@tynandebold would you please create a Kedro Viz project on RTD and add me as maintainer? I'm https://readthedocs.org/profiles/Juanlu001/. I don't have enough permissions to do the automatic configuration, and it's better that we do it that way.

[tynandebold]

I've added you. You should have an invite in your inbox.

[astrojuanlu]

Got access 🙌🏽

[astrojuanlu]

🚀 We shipped support for customizing the URL path for projects and subprojects, allowing you to remove or customize the /projects/ path on subprojects. This is enabled via Support request currently, and only on certain plans on Read the Docs for Business.

https://blog.readthedocs.com/newsletter-july-2023/

[astrojuanlu]

Subprojects for kedro-datasets and kedro-viz are created

• https://docs.kedro.org/projects/kedro-viz/en/latest/
• https://docs.kedro.org/projects/kedro-datasets/en/latest/

Follow up tasks:

• Fix the builds, for which we need to set up distinct Heap properties for each subproject
• Fix the styling so they use exactly the same theme as the Kedro docs, might require refactoring our Sphinx theme into a separate package for easier reuse
• Include links from the subprojects back to Kedro
• Include links from the Kedro docs to the subprojects - these could be in the same left navigation bar we have currently, or a top navigation bar if we happen to tackle Improve the usability of the documentation #4257 first

[stichbury]

Tasks

• Fix the builds, for which we need to set up distinct Heap properties for each subproject Set up subproject builds for new RTD projects #2947
• Fix the styling so they use exactly the same theme as the Kedro docs, might require refactoring our Sphinx theme into a separate package for easier reuse Move Kedro documentation CSS to package allow sharing across subprojects #3016
• Update links in the Kedro docs to the datasets subproject Update content links to subprojects docs when they are published #3015
• Update links in Kedro docs -> Viz docs and likewise Viz docs -> Kedro docs. ticket on Viz project
• Navigation between all Kedro doc assets Interim design changes to accommodate documentation subprojects (datasets & Viz) #3017 (or could be part of Improve the usability of the Framework documentation home page kedro-viz#1474 but I'd prefer to decouple).
• Style datasets docs
• Set up Heap analytics on datasets and viz docs

[astrojuanlu]

• kedro-datasets https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-1.6.0/ ✔️ (but see discussion about the URLs below)
• kedro-viz https://readthedocs.org/projects/kedro-viz/builds/21770555/ ❌ (some dependencies are not in place, but at least the build was attempted)

I'm not very happy with the URL of kedro-datasets though. I've been told that the /projects part can be removed, but the last /kedro-datasets-1.6.0 segment needs to stay, because it comes from the tag name and cannot be changed (readthedocs/readthedocs.org#4167, readthedocs/readthedocs.org#10674). At the moment, to shorten the URLs we would need to resort to workarounds:

• Adapt tag names. But we can't just tag 1.6.0 because kedro-datasets lives in a monorepo with other projects. We could split kedro-datasets to its own repository, but this idea has been met with skepticism in the past. Also, having to do that just because of RTD is weird.
• Ask the RTD team to modify slugs every time. I don't think this is very sustainable and it's not even guaranteed they'll do this for us.
• Accept that URLs will be like this 🤷🏽

[stichbury]

I'm not very happy with the URL of kedro-datasets though. I've been told that the /projects part can be removed, but the last /kedro-datasets-1.6.0 segment needs to stay, because it comes from the tag name and cannot be changed (readthedocs/readthedocs.org#4167, readthedocs/readthedocs.org#10674). At the moment, to shorten the URLs we would need to resort to workarounds:

• Adapt tag names. But we can't just tag 1.6.0 because kedro-datasets lives in a monorepo with other projects. We could split kedro-datasets to its own repository, but this idea has been met with skepticism in the past. Also, having to do that just because of RTD is weird.
• Ask the RTD team to modify slugs every time. I don't think this is very sustainable and it's not even guaranteed they'll do this for us.
• Accept that URLs will be like this 🤷🏽

Can I just check @astrojuanlu why we can't also have stable and latest for datasets as we do for Viz and Framework?

[astrojuanlu]

Because kedro-datasets lives on a monorepo 😓 so all the assumptions RTD makes to create stable and latest break down.

From https://docs.readthedocs.io/en/stable/versions.html

Read the Docs supports multiple versions of your repository. On initial import, we will create a latest version. This will point at the default branch defined in your VCS control (by default, main on Git and default in Mercurial).

If your project has any tags or branches with a name following semantic versioning, we also create a stable version, tracking your most recent release. If you want a custom stable version, create either a tag or branch in your project with that name.

and tags on https://github.com/kedro-org/kedro-plugins/tags do not follow "semantic versioning", because they're prefixed with the plugin in question.

There are no good options here. We either quit RTD over this (and maintain our own infrastructure capable of versioning docs, which is not an easy feat), or we break down the monorepo into separate repos (that's an engineering decision I'd personally be supportive of, and would solve other things like discoverability kedro-org/kedro-plugins#401), or we accept these terrible URLs.

[tynandebold]

My two cents...

We either quit RTD over this (and maintain our own infrastructure capable of versioning docs, which is not an easy feat)

That seems like a non-starter. I don't imagine we'll have capacity to do this anytime soon, if ever (unless there's a quite simple way of achieving this that I'm not seeing).

we break down the monorepo into separate repos

I'm in favor of this. I've found it hard to discover some of our plugins, especially datasets. For example, when I search kedro datasets github on Google, the sixth result is the plugins repo, which isn't exactly what I'm looking for.

And just to throw a wrench in at the end, I don't actually find the URL to be that terrible (https://docs.kedro.org/kedro-datasets/en/kedro-datasets-1.6.0/), though agree that it isn't ideal.

[astrojuanlu]

Thanks @tynandebold. xref kedro-org/kedro-plugins#401 for the discoverability issue.

[stichbury]

The final PRs to address this parent task are sitting in review:

• Framework
• Viz
• Plugins

[stichbury]

These are all now merged 💃 or about to merge. Closing this as (finally) done!