[SPARK-39421] Sphinx build fails with "node class 'meta' is already registered, its visitors will be overridden"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1, 3.3.0, 3.4.0
• Fix Version/s: 3.3.0, 3.2.2, 3.4.0
• Component/s: Documentation
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 09/Jun/22 00:53
• Updated: 12/Dec/22 18:10
• Resolved: 09/Jun/22 05:30

Description

Moving to python/docs directory and building sphinx.
Running Sphinx v3.0.4
WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
/__w/spark/spark/python/pyspark/pandas/supported_api_gen.py:101: UserWarning: Warning: Latest version of pandas(>=1.4.0) is required to generate the documentation; however, your version was 1.3.5
  warnings.warn(
Warning, treated as error:
node class 'meta' is already registered, its visitors will be overridden
make: *** [Makefile:35: html] Error 2
                    ------------------------------------------------
      Jekyll 4.2.1   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------

Sphinx build fails apparently with the latest docutils (see also https://issues.apache.org/jira/browse/FLINK-24662). we should pin the version.

Issue Links

• is duplicated by: SPARK-39424 `Run documentation build` failed in master branch GA
• links to: [Github] Pull Request #36813 (HyukjinKwon)