[SPARK-34943] Upgrade flake8 to 3.8.0 or above in Jenkins

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Shane Knapp
• Reporter: Haejoon Lee
• Created: 02/Apr/21 13:46
• Updated: 12/Dec/22 18:11
• Resolved: 15/Sep/21 00:34

Description

In flake8 < 3.8.0, F401 error occurs for imports in if statements when TYPE_CHECKING is True. However, TYPE_CHECKING is always False at runtime, so there is no need to treat it as an error in static analysis.

Since this behavior is fixed In flake8 >= 3.8.0, we should upgrade the flake8 installed in Jenkins to 3.8.0 or above. Otherwise, it occurs F401 error for several lines in pandas-on-PySpark that uses TYPE_CHECKING.

And also we might update the MINIMUM_FLAKE8 in the lint-python from 3.5.0 to 3.8.0.

Issue Links

• blocks: SPARK-37010 Remove unnecessary "noqa: F401" comments in pandas-on-Spark
• is cloned by: SPARK-37011 Upgrade flake8 to 3.9.0 or above in Jenkins
• links to: [Github] Pull Request #32749 (pingsutw)