Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5590

Create a proper internal state hierarchy

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0
    • Labels:
      None

      Description

      Currently, the state interfaces (like ListState, ValueState, ReducingState) are very sparse and contain only methods exposed to the users. That makes sense to keep the public stable API minimal

      At the same time, the runtime needs more methods for its internal interaction with state, such as:

      • setting namespaces
      • accessing raw values
      • merging namespaces

      These are currently realized by re-creating or re-obtaining the state objects from the KeyedStateBackend. That method causes quite an overhead for each access to the state

      The KeyedStateBackend tries to do some tricks to reduce that overhead, but does it only partially and induces other overhead in the course.

      The root cause of all these issues is a problem in the design: There is no proper "internal state abstraction" in a similar way as there is an external state abstraction (the public state API).

      We should add a similar hierarchy of states for the internal methods. It would look like in the example below:

       *             State
       *               |
       *               +-------------------InternalKvState
       *               |                         |
       *          MergingState                   |
       *               |                         |
       *               +-----------------InternalMergingState
       *               |                         |
       *      +--------+------+                  |
       *      |               |                  |
       * ReducingState    ListState        +-----+-----------------+
       *      |               |            |                       |
       *      +-----------+   +-----------   -----------------InternalListState
       *                  |                |
       *                  +---------InternalReducingState
      

        Issue Links

          Activity

          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in 3b97128f05bacfb80afe4a2a49741c31ff306cd2

          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 3b97128f05bacfb80afe4a2a49741c31ff306cd2
          Hide
          xiaogang.shi Xiaogang Shi added a comment -

          Stephan Ewen, do you have any ideas of the solution?
          I think KvState already provides some needed internal methods. Maybe we can extend it to create the internal state hierarchy?

          Show
          xiaogang.shi Xiaogang Shi added a comment - Stephan Ewen , do you have any ideas of the solution? I think KvState already provides some needed internal methods. Maybe we can extend it to create the internal state hierarchy?

            People

            • Assignee:
              StephanEwen Stephan Ewen
              Reporter:
              StephanEwen Stephan Ewen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development