Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24763

Remove redundant key data from value in streaming aggregation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      Key/Value of state in streaming aggregation is formatted as below:

      • key: UnsafeRow containing group-by fields
      • value: UnsafeRow containing key fields and another fields for aggregation results

      which data for key is stored to both key and value.

      This is to avoid doing projection row to value while storing, and joining key and value to restore origin row to boost performance, but while doing a simple benchmark test, I found it not much helpful compared to "project and join". (will paste test result in comment)

      So I would propose a new option: remove redundant in stateful aggregation. I'm avoiding to modify default behavior of stateful aggregation, because state value will not be compatible between current and option enabled.

        Attachments

          Activity

            People

            • Assignee:
              kabhwan Jungtaek Lim
              Reporter:
              kabhwan Jungtaek Lim
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: