[SPARK-30823] %PYTHONPATH% not set in python/docs/make2.bat, resulting in failed/wrong documentation builds

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.5
• Fix Version/s: 2.4.6
• Component/s: Documentation, PySpark, Windows

Environment

Tested on Windows 10.

• Assignee: David Toneian
• Reporter: David Toneian
• Created: 13/Feb/20 21:06
• Updated: 07/May/20 17:59
• Resolved: 15/Feb/20 22:23

Description

When building the PySpark documentation on Windows, by changing directory to python/docs and running make.bat (which runs make2.bat), the majority of the documentation may not be built if pyspark is not in the default %PYTHONPATH%. Sphinx then reports that pyspark (and possibly dependencies) cannot be imported.

If pyspark is in the default %PYTHONPATH%, I suppose it is that version of pyspark – as opposed to the version found above the python/docs directory – that is considered when building the documentation, which may result in documentation that does not correspond to the development version one is trying to build.

python/docs/Makefile avoids this issue by setting
export PYTHONPATH=$(realpath ..):$(realpath ../lib/py4j-0.10.8.1-src.zip)
on line 10, but make2.bat does no such thing. The fix consist of adding
set PYTHONPATH=..;..\lib\py4j-0.10.8.1-src.zip
to make2.bat.

See GitHub PR #27569.

Issue Links

• links to: GitHub Pull Request #27569