Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39590

Python API Parity in Structure Streaming

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.3.0
    • None
    • Structured Streaming
    • None

    Description

      New APIs in Structured Streaming tend to get added to Java/Scala first.  This creates a situation where the Python API have fallen behind.  For example map/flatMapGroupsWithState is not supported in the Pyspark.  We need Pyspark API to catch up with the Java/Scala APIs and, where necessary, provide tighter integrations with native python data processing frameworks such as Pandas.

      Attachments

        Issue Links

          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

          • Major - Major loss of function.
          • Open
          is duplicated by

          Umbrella - An overarching type made of sub-tasks SPARK-40431 Introduce "Arbitrary Stateful Processing" in Structured Streaming with Python

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              jerrypeng Boyang Jerry Peng
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: