Create redirects automatically on file renames

[humitos]

It would be awesome if Read the Docs could detect that a file was renamed and automatically creates a redirect for this file, so the URLs for the old resource keeps working.

I did a small research about how to get this data from git and I found that it's possible to detect delete/add par files as a rename if more than X percentage of the file hasn't changed.

In the following example, the command shows only those files that haven't change more than 70%. Taking a look at the output, seems accurate to what we have done with our docs.

$ git diff --name-status --diff-filter=R --find-renames=70 8629f62 39de1b7

R080    docs/yaml-config.rst    docs/config-file/v1.rst
R100    docs/architecture.rst   docs/development/architecture.rst
R090    docs/design.rst docs/development/design.rst
R100    docs/design/index.rst   docs/development/design/index.rst
R095    docs/design/theme-context.rst   docs/development/design/theme-context.rst
R091    docs/design/yaml-file.rst       docs/development/design/yaml-file.rst
R096    docs/i18n.rst   docs/development/i18n.rst
R078    docs/install.rst        docs/development/install.rst
R098    docs/issue-labels.rst   docs/development/issue-labels.rst
R100    docs/symlinks.rst       docs/development/symlinks.rst
R074    docs/canonical.rst      docs/guides/canonical.rst
R073    docs/conda.rst  docs/guides/conda.rst
R095    docs/vcs.rst    docs/guides/vcs.rst
R100    readthedocs/restapi/README.rst  readthedocs/api/v2/README.rst

With this data, we could detect which of these files are under a documentation path (defined in config file) and match the documentation type (rst, md) and create a redirect with this data. Example, from docs/yaml-config.rst to docs/config-file/v1.rst.

We need to consider what type of builder the documentation uses (e.g. dirhtml) and build the URL accordingly.

Related to #4221

[ericholscher]

I think this should not be something we build into the platform, but a tool we create for users as a CLI app, that would create redirects against the API. Even better, it could just output the YAML format for our fancy YAML-based redirects that we'll build :)

[humitos]

My idea behind implementing this under RTD is that you can forget about this and leave RTD to handle it for you. Otherwise, each time you rename a file you have to remember to update these redirects to match the current state. So, I'm trying to think in a way where the detection of file changes can either:

1. create the redirects automatically at build time (the problem with this approach is that it will get out-of-sync with YAML file)
2. make the build to fail because there is a filename change that does not have a redirect

I'm not convinced what's the best implementation for this, but whatever it be, I'd love to make the feature let me know about redirects changes instead of myself telling the platform that I changed something.

[Daltz333]

Currently we're developing an extension that solves this
https://github.com/wpilibsuite/sphinxext-rediraffe.

We have this setup as a CI check to determine when a git rename occurs, and then it fails CI if it detects. The contributor then has to add the redirect themselves, or run our autoredirect sphinx builder which adds all 100% prediction redirects to the redirects file. So it's not fully automatic, but we've determined it to be pretty good.

[Daltz333]

Just released a stable release of the extension which includes a builder for CI checking redirect.
The extension:

• Traverses the list of redirects in the redirect file (follows redirects if a redirect redirects to another redirect) so the user only ever gets redirected once.
• Includes a CI check for verifying all files compared to X (our example uses origin/master for comparing changes to CI) have been redirected. If a renamed file has been deleted or renamed, it needs to be added to redirects.
• A builder to automatically add said redirects to the redirects file if the accuracy is greater than the specified value (git reports accuracy based on line numbers changed, a rename with no changes is 100%)
• Also includes a host of other stuff with good test coverage of over 77 tests.

[stsewd]

I had an idea, not related to automatically create redirects, but more related to detect broken pages, so users can take action on that.

We have files in storage, we can use that to compare all files that are missing from a version with another, for example, compare the files from v2 and v3 version. Or even do the comparison on build time, we have the files on the local system, and we can compare the files that are deleted (maybe rclone could provide that information even, maybe in the output, bc didn't find any option for it). Not sure where we could show that information, though.