Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40431 Introduce "Arbitrary Stateful Processing" in Structured Streaming with Python
  3. SPARK-40443

Support applyInPandasWithState in batch query

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Structured Streaming
    • None

    Description

      In Scala/Java, flatMapGroupsWithState API uses different physical plan for batch query, which applies some wrap-up with user function to effectively match up with user function in MapGroupsExec.

      This seems to be non-trivial to deal with in PySpark, or at least seems to require major work. This ticket tracks the effort of supporting the API in batch query.

      Attachments

        Activity

          People

            Unassigned Unassigned
            kabhwan Jungtaek Lim
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: