Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3211 Add AWS Kinesis streaming connector
  3. FLINK-3229

Kinesis streaming consumer with integration of Flink's checkpointing mechanics

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.1.0
    • Connectors / Common
    • None

    Description

      Opening a sub-task to implement data source consumer for Kinesis streaming connector (https://issues.apache.org/jira/browser/FLINK-3211).

      An example of the planned user API for Flink Kinesis Consumer:

      Properties kinesisConfig = new Properties();
      config.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
      config.put(KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_TYPE, "BASIC");
      config.put(
          KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID, 
          "aws_access_key_id_here");
      config.put(
          KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
          "aws_secret_key_here");
      config.put(KinesisConfigConstants.CONFIG_STREAM_INIT_POSITION_TYPE, "LATEST"); // or TRIM_HORIZON
      
      DataStream<T> kinesisRecords = env.addSource(new FlinkKinesisConsumer<>(
          "kinesis_stream_name",
          new SimpleStringSchema(),
          kinesisConfig));
      

      Attachments

        Issue Links

          Activity

            People

              tzulitai Tzu-Li (Gordon) Tai
              tzulitai Tzu-Li (Gordon) Tai
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: