ci: Add docs rtd check on kedro-datasets

[merelcht]

Description

Temporary solution for: #261

Development notes

This approach adds a build that runs:
python -m sphinx -WETan -j auto -D language=en -b linkcheck -d _build/doctrees docs/source _build/linkcheck

This is the check in the read the docs builds that flags syntax issues etc in the docs. It's also the check that has failed and blocked Kedro releases previously, because there were syntax errors in the datasets doc strings that are not caught by the regular linters.

Questions for reviewers

1. Is this check sufficient or would it be better to run the full read the docs build here as well or even producing full docs? The original issue proposes adding docs to this repo, whereas this solution only runs the check to discover syntax issues. UPDATE: I'd suggest getting this in temporarily but still implement the subproject RTD stuff properly and then just removing this build step when that's in place.
2. I copied over the docs config from the core kedro repo and then removed files/lines in conf.py that weren't necessary. I wasn't sure about any files in source/_templates. Are those needed @astrojuanlu / @stichbury ?
3. Any other parts that can potentially be removed, especially from conf.py if we're happy with just running the check instead of producing full docs?

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[stichbury]

Hi @merelcht I am sorry I wasn't more clued in to what you were working on with this as we could have discussed in advance what the overall plan is...which is indeed to move the docs into a RTD subproject per kedro-org/kedro#2600 (which means moving the docs into this repo, creating a new subproject on RTD etc.).

I think what you've done is fine but probably it doesn't go far enough to meet that subproject goal, although I'm still not totally clear because I can't see the RTD subproject (if you've created it, please could you add me as admin https://readthedocs.org/profiles/stichbury/ ?

[merelcht]

Hi @merelcht I am sorry I wasn't more clued in to what you were working on with this as we could have discussed in advance what the overall plan is...which is indeed to move the docs into a RTD subproject per kedro-org/kedro#2600 (which means moving the docs into this repo, creating a new subproject on RTD etc.).

I think what you've done is fine but probably it doesn't go far enough to meet that subproject goal, although I'm still not totally clear because I can't see the RTD subproject (if you've created it, please could you add me as admin https://readthedocs.org/profiles/stichbury/ ?

My approach now doesn't move the docs into a RTD subproject. Instead I went for a quick fix approach to make sure that for the coming releases we don't get blocked again by syntax problems in kedro-datasets. This solution only runs python -m sphinx -WETan -j auto -D language=en -b linkcheck -d _build/doctrees docs/source _build/linkcheck on the kedro-datasets "docs", which are basically just the docstrings.

I'd suggest getting this in temporarily but then when the RTD subproject has been setup properly remove this build step again. I assumed that the RTD setup will take more time and I wanted to fix this blocker quickly 😄

[astrojuanlu]

Setting up RTD might actually take less 😄 in any case, the initial work of doing sphinx-quickstart seems to be already done, so whether the sphinx-build command runs on GHA or RTD doesn't matter. But eventually we'll want an RTD project so that we can combine the docs of both projects.

I'd take care of this myself but I don't have permissions. Will try to review the current approach locally in the meantime.

[merelcht]

I'd take care of this myself but I don't have permissions. Will try to review the current approach locally in the meantime.

What do you need permission for? Maybe I can grant it 😃

[astrojuanlu]

I gave this a quick try locally, found this error after installing the dependencies:

$ cd kedro-datasets/docs && sphinx-build -WETan -j auto -D language=en -b linkcheck -d _build/doctrees source _build/linkcheck
[...looooong list of DeprecationWarning...]
[08/14/23 13:30:16] WARNING  /Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/ warnings.py:109
                             site-packages/kedro_datasets/yaml/yaml_dataset.py:10:                         
                             DeprecationWarning: 'DataSetError' has been renamed to                        
                             'DatasetError', and the alias will be removed in Kedro 0.19.0                 
                               from kedro.io.core import (                                                 
                                                                                                           

Traceback (most recent call last):
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/generate.py", line 395, in generate_autosummary_docs
    name, obj, parent, modname = import_by_name(entry.name)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/__init__.py", line 661, in import_by_name
    raise ImportExceptionGroup('no module named %s' % ' or '.join(tried), exceptions)
sphinx.ext.autosummary.ImportExceptionGroup: no module named kedro_datasets.pandas.DeltaTableDataSet

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/generate.py", line 400, in generate_autosummary_docs
    name, obj, parent, modname = import_ivar_by_name(entry.name)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/__init__.py", line 732, in import_ivar_by_name
    raise ImportError
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/application.py", line 262, in __init__
    self._init_builder()
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/application.py", line 335, in _init_builder
    self.events.emit('builder-inited')
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/__init__.py", line 811, in process_generate_options
    generate_autosummary_docs(genfiles, suffix=suffix, base_path=app.srcdir,
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/ext/autosummary/generate.py", line 409, in generate_autosummary_docs
    logger.warning(__('[autosummary] failed to import %s.\nPossible hints:\n%s'),
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1847, in warning
    self.log(WARNING, msg, *args, **kwargs)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/util/logging.py", line 123, in log
    super().log(level, msg, *args, **kwargs)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1879, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1547, in log
    self._log(level, msg, args, **kwargs)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 964, in handle
    rv = self.filter(record)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/logging/__init__.py", line 821, in filter
    result = f.filter(record)
  File "/Users/juan_cano/.micromamba/envs/kedro310-dev/lib/python3.10/site-packages/sphinx/util/logging.py", line 426, in filter
    raise exc
sphinx.errors.SphinxWarning: [autosummary] failed to import kedro_datasets.pandas.DeltaTableDataSet.
Possible hints:
* ImportError: 
* ModuleNotFoundError: No module named 'kedro_datasets.pandas.DeltaTableDataSet'
* AttributeError: module 'kedro_datasets.pandas' has no attribute 'DeltaTableDataSet'

Warning, treated as error:
[autosummary] failed to import kedro_datasets.pandas.DeltaTableDataSet.
Possible hints:
* ImportError: 
* ModuleNotFoundError: No module named 'kedro_datasets.pandas.DeltaTableDataSet'
* AttributeError: module 'kedro_datasets.pandas' has no attribute 'DeltaTableDataSet'

This is probably related to kedro-org/kedro#2871 (comment), we never opened an issue on this side I think.

After I did pip install ./kedro-datasets[all,pandas-deltatabledataset], I tried again and complained about kedro_datasets.snowflake.SnowparkTableDataSet.

[astrojuanlu]

For the record, the kedro-datasets project now exists on Read the Docs! With your permission @merelcht I'll push a .readthedocs.yml file to this branch. I'm not sure if we'll see the RTD check appear here (maybe we'll need to merge first)

[astrojuanlu]

RTD builds are on now: https://readthedocs.org/projects/kedro-plugins/builds/21606215/

[astrojuanlu]

Aaaaand it worked! https://kedro-plugins--299.org.readthedocs.build/en/299/kedro_datasets.html

[merelcht]

Both your links return 404 for me @astrojuanlu 🤔 But I can see the checks passed!

[stichbury]

Happy to approve this. Thanks @merelcht!

[astrojuanlu]

I requested a rename and it happened overnight, now it's https://readthedocs.org/projects/kedro-datasets/builds/21613210/

[SajidAlamQB]

🚀

[noklam]

What's the two different RTD build? One is giving me 404 error.

[astrojuanlu]

Project URL changed from kedro-plugins to kedro-datasets