Manually create a sitemap.xml for docs SEO

[DimedS]

Description

We initially tried using autogenerated robots.txt and sitemap.xml. However, the issue with this approach is that it does not support sub-projects (Viz and Datasets).

This PR reverts to using a manual robots.txt, with a link to a manually created sitemap.xml. Both files are now located in the extra_files folder, and this path has been included in the html_extra_path configuration in conf.py.

If this approach proves successful, the next step will be to completely deindex older docs versions using the method proposed by RTD: readthedocs/readthedocs.org#10648 (comment).

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[DimedS]

After adding html_extra_path = [str(here / "sitemap.xml")] to conf.py, I observed the following post-build:

• The custom sitemap.xml is accessible at: https://kedro--4261.org.readthedocs.build/en/4261/sitemap.xml
• The autogenerated sitemap.xml is still located at: https://kedro--4261.org.readthedocs.build/sitemap.xml and is likely the version that will be used at https://docs.kedro.org/sitemap.xml.

I also tried adding a custom robots.txt, but the autogenerated version still appears at https://kedro--4261.org.readthedocs.build/robots.txt. This makes it difficult to verify the effect in this build, and it’s possible that the custom sitemap.xml and robots.txt will only take effect at a later stage, potentially being used at https://docs.kedro.org.

[astrojuanlu]

💯