Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35800

Improving testability of GroupState in streaming flatMapGroupsWithState

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None
    • Structured Streaming
    • None

    Description

      GroupStateImpl is the internal implementation of the GroupState interface which mean to be not exposed. Thus, it only has a private constructor. Such access control does benefit encapsulation, however, this introduces difficulties for unit tests and the users are calling the engine to construct such GroupState instances in order to test their customized state transition functions.

      The solution is to introduce new interfaces that allow users to create instances of GroupState but also access internal values of what they have set (for example, has to state been updated, or removed). This would allow them to write unit tests of the state transition function with custom GroupState objects and then verifying whether the state was updated in an expected way.

      Attachments

        Activity

          People

            Unassigned Unassigned
            tdas Tathagata Das
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: