Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17972

Consider restructuring channel state

    XMLWordPrintableJSON

Details

    Description

      (depends on rescaling for unaligned checkpoints (FLINK-17979))

       

      Current structure is the following (this PR doesn't change it):

      Each subtask reports to JM TaskStateSnapshot
        each with zero ore more OperatorSubtaskState,
          each with zero or more InputChannelStateHandle and ResultSubpartitionStateHandle
            each referencing an underlying StreamStateHandle
      

      The underlying StreamStateHandle duplicates filename (ByteStreamStateHandle has it too at least because of equals/hashcode I guess).

      An alternative would be something like

      Each subtask reports to JM TaskStateSnapshot
        each with zero ore more OperatorSubtaskState
          each with zero or one StreamStateHandle (for channel state)
          each with zero or more InputChannelStateHandle and ResultSubpartitionStateHandle

      (probably, with StreamStateHandle and InputChannelStateHandle and ResultSubpartitionStateHandle encapsulated)

       

      It would be more effective (less data duplication) but probably also more error-prone (implicit structure), less flexible (re-scaling).

      (as discussed during introduction of StreamStateHandle.asBytesIfInMemory here)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              roman Roman Khachatryan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: