[RTD] Backward-compatibility code trips documentation builds on downstream projects

[amotl]

Problem

The recently added RTD backward-compatibility code in src/crate/theme/vendor/rtd_compat/extension.py::manipulate_config is highly problematic when it comes to TimeoutErrors, as it emits warnings then, which trip the build, because they are treated as errors in our setting.

crate-docs-theme/src/crate/theme/vendor/rtd_compat/extension.py

Lines 14 to 131 in 0644241

	# https://www.sphinx-doc.org/en/stable/extdev/appapi.html#event-html-page-context
	def manipulate_config(app, config):
	logger.info(
	'Running "manipulate_config" from Read the Docs "sphinx_build_compatibility" extension. '
	'Consider removing it from your requirements and migrating your documentation accordingly. '
	'This extension is useful only as a transition but it will not be maintained.'
	)
	# Add Read the Docs' static path.
	# Add to the end because it overwrites previous files.
	if not hasattr(config, "html_static_path"):
	config.html_static_path = []
	if os.path.exists('_static'):
	config.html_static_path.append('_static')
	# Define this variable in case it's not defined by the user.
	# It defaults to `alabaster` which is the default from Sphinx.
	# https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_theme
	if not hasattr(config, "html_theme"):
	config.html_theme = 'alabaster'
	# Example: ``/docs/``
	conf_py_path = "/"
	conf_py_path += os.path.relpath(
	str(app.srcdir),
	os.environ.get("READTHEDOCS_REPOSITORY_PATH"),
	).strip("/")
	conf_py_path += "/"
	git_clone_url = os.environ.get("READTHEDOCS_GIT_CLONE_URL")
	github_user, github_repo = get_github_username_repo(git_clone_url)
	bitbucket_user, bitbucket_repo = get_bitbucket_username_repo(git_clone_url)
	gitlab_user, gitlab_repo = get_gitlab_username_repo(git_clone_url)
	project_slug = os.environ.get("READTHEDOCS_PROJECT")
	version_slug = os.environ.get("READTHEDOCS_VERSION")
	production_domain = os.environ.get("READTHEDOCS_PRODUCTION_DOMAIN", "readthedocs.org")
	scheme = "https"
	if production_domain.startswith("devthedocs"):
	scheme = "http"
	# We are using APIv2 to pull active versions, downloads and subprojects
	# because APIv3 requires a token.
	try:
	response_project = requests.get(
	f"{scheme}://{production_domain}/api/v3/projects/{project_slug}/",
	timeout=2,
	).json()
	language = response_project["language"]["code"]
	except Exception:
	logger.warning(
	"An error ocurred when hitting API to fetch project language. Defaulting to 'en'.",
	exc_info=True,
	)
	language = "en"
	try:
	response_versions = requests.get(
	f"{scheme}://{production_domain}/api/v3/projects/{project_slug}/versions/?active=true",
	timeout=2,
	).json()
	versions = [
	(version["slug"], f"/{language}/{version['slug']}/")
	for version in response_versions["results"]
	]
	except Exception:
	logger.warning(
	"An error ocurred when hitting API to fetch active versions. Defaulting to an empty list.",
	exc_info=True,
	)
	versions = []
	try:
	downloads = []
	for version in response_versions["results"]:
	if version["slug"] != version_slug:
	continue
	for key, value in version["downloads"]:
	downloads.append(
	(
	key,
	value,
	),
	)
	except Exception:
	logger.warning(
	"An error ocurred when generating the list of downloads. Defaulting to an empty list.",
	exc_info=True,
	)
	downloads = []
	try:
	subprojects = []
	response_project = requests.get(
	f"{scheme}://{production_domain}/api/v2/project/?slug={project_slug}",
	timeout=2,
	).json()
	project_id = response_project["results"][0]["id"]
	response_subprojects = requests.get(
	f"{scheme}://readthedocs.org/api/v2/project/{project_id}/subprojects/",
	timeout=2,
	).json()
	for subproject in response_subprojects["subprojects"]:
	subprojects.append(
	(
	subproject["slug"],
	subproject["canonical_url"],
	),
	)
	except Exception:
	logger.warning(
	"An error ocurred when hitting API to fetch project/subprojects. Defaulting to an empty list.",
	exc_info=True,
	)
	subprojects = []

References

WARNING: An error ocurred when hitting API to fetch active versions. Defaulting to an empty list.
WARNING: An error ocurred when generating the list of downloads. Defaulting to an empty list.
build finished with problems, 2 warnings.

• https://readthedocs.org/projects/crate/builds/26356993/

[msbt]

@amotl is that related to the issue that you were discussing on Slack with @BaurzhanSakhariev?

[amotl]

I don't think it is related to this specific issue. While it might be related to general refurbishments and modernizations at RTD, the other thing is related to networking and connectivity issues, and this one is related to Sphinx, so effectively, totally different domains.

[amotl]

An error ocurred when hitting API to fetch active versions.

Ah, right, both are about rejected requests, possibly by rate limiting machineries. So, well, yeah, in this case both issues are probably connected.

[amotl]

Maybe we need to tell them about the IP address of our Jenkins server?

[amotl]

Maybe we need to tell them about the IP address of our Jenkins server?

Hm, not necessarily. In this case, RTD is tripping itself: https://readthedocs.org/projects/crate/builds/26356993/

[amotl]

I've now also observed it happening on my workstation, when working on cratedb-guide in sandbox mode.

Running Sphinx v7.1.2

Running "manipulate_config" from Read the Docs "sphinx_build_compatibility" extension. Consider removing it from your requirements and migrating your documentation accordingly. This extension is useful only as a transition but it will not be maintained.

Warning, treated as error:
An error ocurred when hitting API to fetch active versions. Defaulting to an empty list.
Command exited with exit code: 2