Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6653

Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 1.4.0
    • Component/s: Kinesis Connector
    • Labels:
      None

      Description

      Currently, the Kinesis consumer's checkpoints directly serialize AWS's Shard instances in checkpoints. This makes bumping AWS library versions hard, since any change to the Shard class by AWS will break checkpoint compatibility.

      We should either have custom serialization for KinesisStreamShard, or disintegrate the information in Shard. Ideally, it would be best to make KinesisStreamShard and SequenceNumber to be non-serializable, hence avoiding Java serialization in the checkpoints.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tonywei Wei-Che Wei
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: