Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40431 Introduce "Arbitrary Stateful Processing" in Structured Streaming with Python
  3. SPARK-40581

Improving testability of GroupState in applyInPandasWithState

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Structured Streaming
    • None

    Description

      SPARK-35800 introduced a testable instance of GroupState in Scala/Java API. The rationalization was that default implementation of GroupState couldn't be instantiated.

      The GroupState being introduced in PySpark is something end users can create an instance directly, but looks like the constructor is only accounting the case it comes from serialized form from executor.

      It would be better to enable GroupState instance to be created by end user for testing purpose, with isolation to the constructor for internal usage.

      Attachments

        Activity

          People

            Unassigned Unassigned
            kabhwan Jungtaek Lim
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: