[SPARK-16992] Pep8 code style

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: gsemet
• Created: 10/Aug/16 09:00
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

Add code style checks and auto formating into the Python code.

Features:

• add a .editconfig file (Spark's Scala files use 2-spaces indentation, while Python files uses 4) for compatible editors (almost every editors has a plugin to support .editconfig file)
• use autopep8 to fix basic pep8 mistakes
• use isort to automatically sort and organise import statements and organise them into logically linked order (see doc here. The most important thing is that it splits import statements that loads more than one object into several lines. It send keep the imports sorted. Said otherwise, for a given module import, the line where it should be added will be fixed. This will increase the number of line of the file, but this facilitates a lot file maintainance and file merges if needed.
  add a 'validate.sh' script in order to automatise the correction (need isort and autopep8 installed)
  You can see similar script in prod in the Buildbot project.

Issue Links

• is related to: SPARK-19002 Check pep8 against all the python scripts
• relates to: SPARK-16367 Wheelhouse Support for PySpark
• links to: [Github] Pull Request #14567 (Stibbons)
• links to: [Github] Pull Request #14830 (Stibbons)
• links to: [Github] Pull Request #14863 (Stibbons)
• links to: [Github] Pull Request #14963 (Stibbons)