[FLINK-24229] DynamoDB implementation of Async Sink

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: aws-connector-3.0.0
• Component/s: Connectors / DynamoDB
• Labels: pull-request-available
• Assignee: Yuri Gusev
• Reporter: Zichen Liu
• Created: 09/Sep/21 12:45
• Updated: 01/May/24 15:13
• Resolved: 04/Nov/22 22:06

Description

Motivation

User stories:
As a Flink user, I’d like to use DynamoDB as sink for my data pipeline.

Scope:

• Implement an asynchronous sink for DynamoDB by inheriting the AsyncSinkBase class. The implementation can for now reside in its own module in flink-connectors.
• Implement an asynchornous sink writer for DynamoDB by extending the AsyncSinkWriter. The implementation must deal with failed requests and retry them using the requeueFailedRequestEntry method. If possible, the implementation should batch multiple requests (PutRecordsRequestEntry objects) to Firehose for increased throughput. The implemented Sink Writer will be used by the Sink class that will be created as part of this story.
• Java / code-level docs.
• End to end testing: add tests that hits a real AWS instance. (How to best donate resources to the Flink project to allow this to happen?)

References

More details to be found https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink

Issue Links

• causes: FLINK-25859 Documentation for DynamoDB Sink
• Dependent: FLINK-24041 [FLIP-171] Generic AsyncSinkBase
• is a parent of: FLINK-29895 Improve code coverage and integration tests for DynamoDB implementation of Async Sink
• is cloned by: FLINK-25731 Mark FlinkKinesisProducer/FlinkKinesisConsumer as deprecated
• supercedes: FLINK-16504 Add a AWS DynamoDB sink
• links to: GitHub Pull Request #1
• links to: GitHub Pull Request #1
• links to: GitHub Pull Request #18518
• mentioned in: Page Loading...