Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.12.0, 1.12.1, 1.12.2, 1.13.0, 1.12.3, 1.13.1, 1.12.4, 1.12.5, 1.13.2
Description
Background
The Kinesis EFO consumer uses an async AWS SDK Netty client to read records from Kinesis. When the client is inactive for 30 seconds a ReadTimeoutException is thrown by Netty. The consumer will terminate the subscription, backoff and retry. Jobs with high backpressure can result in frequent ReadTImeoutException and the frequent backoff and retry can cause unnecessary overhead.
What?
Reduce/eliminate ReadTimeoutException from the EFO consumer
How?
There are 2 improvements to be made:
1. Request next record from the Flink source thread rather than the AWS SDK response thread. This means that there will always be space in the input buffer queue. The AWS SDK async response thread is no longer blocking on this queue. Backpressure is now applied by the Flink source thread rather than the AWS SDK thread.
2. Increase the Read Timeout (30s) to be higher than the maximum Shard subscription duration (5m) and enable TCP keep alive
References
This has already been implemented and tested in amazon-kinesis-connector-flink: