Top-level documentation of namespace packages?

[jkbecker]

Expected Behavior

My project consists of multiple modules packaged in a namespace package (because some additional sub-modules are managed in separate repos). I would like to add a high-level introductory chapter to my documentation (i.e., on the "home page" of the HTML output). I am running pdoc3 --output-dir docs --html packagedir to generate documentation.

Actual Behavior

When I run the command above, I'm getting documentation on all sub-modules, but I would like to import a Markdown file above the "Sub-modules" headline.

Is this possible somehow?

Steps to Reproduce

N/A (see above)

Additional info

• pdoc version: 0.9.2

[kernc]

For the duration of building the docs, can you write an __init__.py file with roughly the contents:

# namespace/__init__.py:
""".. include:: relative/path/to.md"""

using the .. include:: directive?

Does this work for namespace packages? 🤔

[jkbecker]

Interesting hack, I'll look into it. It's certainly possible (in the sense that I fully control the project and don't mind wrapping docs building in a shell script) but I'll need to see if it produces the intended results (it will not be recognized as a namespace package any more most likely, but that's actually OK for documentation purposes in my case).
Will report back!

[jkbecker]

Ok, I tried this and it doesn't work for me: The problem here is that if I add an __init__.py to my namespace directory, the one sub-module that is defined elsewhere will be seen as missing during pdoc's runtime, i.e., I get a ModuleNotFoundError for that sub-module and documentation fails.

My only other idea is to run pdoc without the inserted __init__.py file which will give me a correct documentation of the entire installed namespace package (with a bland table of content and no intro as shown above), and then patch the top-level index.html by inserting content from a designated main Markdown file... The thing is, I don't want to do this manually either... could someone point me to the relevant backend code that lets me use pdoc3 functionality to generate the right HTML from a given Markdown file so I can patch it in semi-conveniently?

My ideal scenario would be pdoc3support for something like pyproject.toml or setup.cfg to import some top level documentation in the absence of an __init__.py file, but I assume this is a bit much to ask on the spot 😬
Has supporting either of these files ever been discussed for pdoc3?

[kernc]

Currently, to_html() is used for docstring-to-HTML conversion. module= is the active module acting, primarily, as a source of context while link= is a hyperlinking function such as:

pdoc/pdoc/templates/html.mako

Lines 8 to 14 in 72e41db

	def link(dobj: pdoc.Doc, name=None):
	name = name or dobj.qualname + ('()' if isinstance(dobj, pdoc.Function) else '')
	if isinstance(dobj, pdoc.External) and not external_links:
	return name
	url = dobj.url(relative_to=module, link_prefix=link_prefix,
	top_ancestor=not show_inherited_members)
	return f'<a title="{dobj.refname}" href="{url}">{name}</a>'

to import some top level documentation in the absence of an init.py file

This sounds fairly specific. Including extra pages/content has been discussed before: #233, #115. I don't think anyone's been considering the files — we kinda use our own config file:

pdoc/pdoc/templates/config.mako

Lines 1 to 3 in 72e41db

	<%!
	# Template configuration. Copy over in your template directory
	# (used with `--template-dir`) and adapt as necessary.

One alternative way would be to just use the submodules' generated docs and construct the index/intro page from scratch.

[jkbecker]

Ah, I was unaware of that config file 👍 . Could there be a homepage (or maybe readme) entry in the config.mako which would be None by default to fall back to current behavior, but could contain a reference to README.md (or any other Markdown file) that would then be included on the main index file? I feel like this would fix #233 as well (#115 is a whole different animal IMO).

I'm happy to work on a PR for this, but I will probably solve my issue in a different way if this doesn't sound like a good idea to you.

[kernc]

I don't vehemently oppose it, but:

main index file?

pdoc can be run as pdoc ... MODULE [MODULE...], i.e. pdoc --html pkg1 pkg2 unrelated_module.py, resulting in:

html
├── pkg1
│   ├── a
│   │   ├── index.html
│   │   └── mod.html
│   └── index.html
├── pkg2
│   ├── b
│   │   ├── index.html
│   │   └── mod2.html
│   └── index.html
└── unrelated_module.html

Which would be the main index file?

Can we think of a generic, customizable solution (such as passing a dict or a callable)?

[jkbecker]

Ah, I understand. In that case, my understanding of such a "main index file" would be one that sits here:

html
├── pkg1
│   ├── a
│   │   ├── index.html
│   │   └── mod.html
│   └── index.html
├── pkg2
│   ├── b
│   │   ├── index.html
│   │   └── mod2.html
│   └── index.html
├── index.html <-------------- HERE
└── unrelated_module.html

And looks somewhat like this:

- imported README.md as specified in `config.mako:readme` (only IF specified!)
- pgk1 TOC link
- pkg2 TOC link
- undelated_module TOC link

In other words, it would be a list of the documented modules that were passed (similar to how it documents namespace packages in my OP example) plus the explicitly designated readme doc at the top.

Just to be clear, I don't want my idea to change default behavior, just provide a way to optionally insert an introductory .md file in the absence of a __init__.py file that could contain the include directive -- this seems to be the case for both my situation as well as the pdoc pkg1 pkg2 ... case.

[jkbecker]

Just wanted to add: I'm curious by the potential to add this as it would benefit my direct use case, but I'm not going to be pushy about it if this is out of scope / too edge-casey for your current roadmap.

I definitely prefer fixing my own documentation with a bash hack than insisting on a feature that doesn't make sense, but happy to contribute it if we can hash it out in a way that seems generally useful :)

[kernc]

So when running pdoc --html mypackage, the main index would still be:

html
├── mypackage
│   ...
└── index.html  <------------- HERE

The list of modules is then kind of like what --http provides answering root /, and a duplicate of #101? The latter issue sure is something I'd care to see closed.

Otherwise, I'm a fan of simple bash recipes. Their span and versatility is unmatched. 😃

[jkbecker]

Ah, interesting discussion over there, I like those drafts...

I patched my urgent needs with the following code for now

docs/doc.sh:

#!/bin/bash
pdoc3 --output-dir docs --html --force mymodule
python3 docs/doc-postprocess.py

docs/doc-postprocess.py:

import markdown

INTRO = 'docs/introduction.md'
INDEX = 'docs/snout/index.html'
INSERT_AT = '<section id="section-intro">'

if __name__ == "__main__":
	with open(INTRO, 'r') as f:
		intro = f.read()
	with open(INDEX, 'r') as f:
		index = f.read()
	with open(INDEX, 'w') as f:
		f.write(index.replace(INSERT_AT, f'{INSERT_AT}\n{markdown.markdown(intro)}'))

Which works fairly well for my specific case.

I'm interested in #101 though -- seems like it's kinda orphaned, does that mean you want someone to look into it and flesh this feature out some more?

[kernc]

Absolutely; if you can prepare a PR, I'd be happy to review it and work towards merging it.

Let's call this a duplicate of #101 then.

[patrickhulce]

If anyone else is struggling with this and wants another option, my solution without any code modifications was to put my toplevel documentation in an additional namespace package called my_namespace.docs, install all the wheels in an isolated venv, then just mv site-packages/my_namespace/docs/* site-packages/my_namespace && pdoc site-packages/my_namespace when I publish to our docsite