Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5932

Order of legacy vs new state initialization in the AbstractStreamOperator.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0
    • Component/s: DataStream API
    • Labels:
      None

      Description

      Currently in the AbstractStreamOperator::initializeState(OperatorStateHandles stateHandles), the restoreStreamCheckpointed which is responsible for restoring state from previous Flink versions, (backwards compatibility) is called before the initializeState(StateInitializationContext context) which is responsible for initializing the state in Flink 1.2.

      This has the negative side effect that when implementing the backwards compatibility strategy for a given operator, we have to restore the old state, store it in local variables, and register it with the new state abstractions in the initializeState() or the open(). This creates a lot of unnecessary code in the operators, and potential memory leaks if the local variables are not "null-ified".

      This issue proposes to call the restoreStreamCheckpointed after the initializeState(StateInitializationContext context). This way, the new operator state will have been initialized (e.g. keyed state), and the restoreStreamCheckpointed will be able to register its state directly with the new abstractions, instead of putting it in local variables and wait for the initializeState or the open() to re-register it.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          aljoscha Aljoscha Krettek added a comment -

          +1 I think this makes a lot of sense

          Stefan Richter what do you think?

          Show
          aljoscha Aljoscha Krettek added a comment - +1 I think this makes a lot of sense Stefan Richter what do you think?
          Hide
          srichter Stefan Richter added a comment -

          I think it makes a lot of sense if it is easily possible.

          Show
          srichter Stefan Richter added a comment - I think it makes a lot of sense if it is easily possible.
          Hide
          kkl0u Kostas Kloudas added a comment -

          Merged with d998ff9175ec84aa5685fbd6ebe79341a4da8e8f

          Show
          kkl0u Kostas Kloudas added a comment - Merged with d998ff9175ec84aa5685fbd6ebe79341a4da8e8f

            People

            • Assignee:
              kkl0u Kostas Kloudas
              Reporter:
              kkl0u Kostas Kloudas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development