Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20611

Support 2 phase local aggregate for window aggregate operation

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      In our company, many jobs run with window aggregate and suffer from large amounts of intermediate shuffle. If we can enable 2 phase local agg on window aggregate, the performance will be improved.

      This issue aims to support 2 phase local agg in window aggregate operator both in SQL and Table API. Simple tumble window will be supported in the first place.

       

      We still use below configurations, but they take effect only inside each single window.

      table.exec.mini-batch.enabled

      table.exec.mini-batch.allow-latency

      table.exec.mini-batch.size

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                maosuhan Suhan Mao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: