[FLINK-3229] Kinesis streaming consumer with integration of Flink's checkpointing mechanics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Connectors / Common
• Assignee: Tzu-Li (Gordon) Tai
• Reporter: Tzu-Li (Gordon) Tai
• Created: 13/Jan/16 08:06
• Updated: 29/May/16 04:52
• Resolved: 18/May/16 13:06

Description

Opening a sub-task to implement data source consumer for Kinesis streaming connector (https://issues.apache.org/jira/browser/FLINK-3211).

An example of the planned user API for Flink Kinesis Consumer:

Properties kinesisConfig = new Properties();
config.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
config.put(KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_TYPE, "BASIC");
config.put(
    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID, 
    "aws_access_key_id_here");
config.put(
    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    "aws_secret_key_here");
config.put(KinesisConfigConstants.CONFIG_STREAM_INIT_POSITION_TYPE, "LATEST"); // or TRIM_HORIZON

DataStream<T> kinesisRecords = env.addSource(new FlinkKinesisConsumer<>(
    "kinesis_stream_name",
    new SimpleStringSchema(),
    kinesisConfig));

Issue Links

• links to: GitHub Pull Request #1911