[SPARK-37570] mypy breaks on pyspark.pandas.plot.core.Bucketizer

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: PySpark

Environment

Mypy: v0.910-1
Py: 3.9

• Assignee: Unassigned
• Reporter: Rafal Wojdyla
• Created: 08/Dec/21 01:13
• Updated: 12/Dec/22 18:10

Description

Mypy breaks on a project with pyspark 3.2.0 dependency (worked fine for 3.1.2), see stacktrace below:

Traceback (most recent call last):
  File "/Users/rav/.cache/pre-commit/repoytd0jvc_/py_env-python3.9/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/rav/.cache/pre-commit/repoytd0jvc_/py_env-python3.9/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 87, in main
  File "mypy/main.py", line 165, in run_build
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 254, in _build
  File "mypy/build.py", line 2697, in dispatch
  File "mypy/build.py", line 3021, in process_graph
  File "mypy/build.py", line 3138, in process_stale_scc
  File "mypy/build.py", line 2288, in write_cache
  File "mypy/build.py", line 1475, in write_cache
  File "mypy/nodes.py", line 313, in serialize
  File "mypy/nodes.py", line 3149, in serialize
  File "mypy/nodes.py", line 3083, in serialize
AssertionError: Definition of pyspark.pandas.plot.core.Bucketizer is unexpectedly incomplete