Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14862

Fuse initalizeState and open passes in StreamTask#invoke

    XMLWordPrintableJSON

    Details

      Description

      Currently chained operators lack the ability to send records downstream during initializeState (AbstractStreamOperator#initalizeState), because downstream operators in the chain didn't open yet (AbstractStreamOperator#open).

      The main problem with that is that the rawOperatorState and the rawKeyedStateInputs are only valid during the execution of initializeState, and with large raw states materializing to hand them over to be used at open is not feasible (at least without spilling to disk)

      So, I would propose to fuse the loops here:

      https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L435,L436

      Into a single loop that calls initalizeState and immediately open.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                igal Igal Shilman
                Reporter:
                igal Igal Shilman
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m