Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7366

Upgrade kinesis producer library in flink-connector-kinesis

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.2
    • Fix Version/s: 1.4.0
    • Component/s: Kinesis Connector
    • Labels:
      None

      Description

      We need to upgrade KPL and KCL to pick up the enhanced performance and stability for Flink to work better with Kinesis. Upgrading KPL is specially necessary, because the KPL version Flink uses is old, and doesn't have good retry and error handling logic.

      KPL:
      flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus problematic. It doesn't even have good retry logic, therefore Flink fails really frequently (about every 10 mins as we observed) when Flink writes too fast to Kinesis and receives RateLimitExceededException,

      Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56, "With the newer version of the KPL it uses the AWS C++ SDK which should offer additional retries." on Oct 2016. 0.12.5, the version we are upgrading to, is released in May 2017 and should have the enhanced retry logic.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                phoenixjiangnan Bowen Li
                Reporter:
                phoenixjiangnan Bowen Li
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: