Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32681

PySpark type hints support

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Labels:
      None

      Description

      https://github.com/zero323/pyspark-stubs demonstrates a lot of benefits to improve usability in PySpark by leveraging Python type hints.

      By having the type hints in PySpark we can, for example:

      • automatically document the input and output types
      • leverage IDE for error detection and auto-completion
      • have a cleaner definition and easier to understand.

      This JIRA is an umbrella JIRA that targets to port https://github.com/zero323/pyspark-stubs and related items to smoothly run within PySpark.

      It was also discussed in the dev mailing list: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html

        Attachments

          Activity

            People

            • Assignee:
              zero323 Maciej Szymkiewicz
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: