Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6653

Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0, 1.4.0
    • Connectors / Kinesis
    • None

    Description

      Currently, the Kinesis consumer's checkpoints directly serialize AWS's Shard instances in checkpoints. This makes bumping AWS library versions hard, since any change to the Shard class by AWS will break checkpoint compatibility.

      We should either have custom serialization for KinesisStreamShard, or disintegrate the information in Shard. Ideally, it would be best to make KinesisStreamShard and SequenceNumber to be non-serializable, hence avoiding Java serialization in the checkpoints.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-5898 Race-Condition with Amazon Kinesis KPL

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #3994

          Activity

            People

              tonywei Wei-Che Wei
              tzulitai Tzu-Li (Gordon) Tai
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: