[BEAM-12235] Python API reference docs need better organization

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: website
• Assignee: Unassigned
• Reporter: Stephan Hoyer
• Created: 27/Apr/21 18:31
• Updated: 04/Jun/22 20:11

Description

The first page of the reference API docs is helpful:
https://beam.apache.org/releases/pydoc/2.28.0/index.html

But diving in, I see the raw internal structure of Beam's Python modules, e.g.,
https://beam.apache.org/releases/pydoc/2.28.0/apache_beam.io.html

This is not very usable:

• It's hard to navigate. I need to know exactly where a function is defined to find it. E.g., to find beam.Map, I had to click on "apache_beam.transforms package" followed by "apache_beam.transforms.core module" and then scroll down or search in the page for "Map."
• It isn't clear exactly which components are public APIs. The documentation for few modules notes that they are not public, but there are so many others listed that I'm sure they cannot all be intended for public support. This makes it hard to find Beam's main public APIs.
• It isn't clear the preferred import paths to use. For example, apache_beam.Map is documented as apache_beam.transforms.core.Map, without mention of the shorter name.

I suspect the source of most of these issues is that the API docs make heavy use of Sphinx's autodoc for modules. In my experience maintaining Python projects, this just doesn't work very. autosummary and autofunction on individual functions/classes work well, but it needs to be organized by hand – you can't count on automodule to do a good job of high level organization. JAX's docs are a good example, e.g., see https://github.com/google/jax/blob/master/docs/jax.rst and https://jax.readthedocs.io/en/latest/jax.html

Happy to advise offline as well – feel free to ping me at [e-mail]