Details

      Description

      Background

      In order to add support for EFO in the FlinkKinesisConsumer we are abstracting out the record consumption from Kinesis ShardConsumer and introducing an interface. 

      Scope

      Introduce the RecordPublisher interface and refactor the existing polling implementation to implement it:

      • Add PollingRecordPublisher that is functionally equivalent to the existing implementation
      • Support adaptive throughput via an extension, AdaptivePollingRecordSubscriber
      • Split out the ShardMetricReporter into separate classes such that each component can report it's own metrics:
        • ShardConsumer
        • PollingRecordConsumer
        • FanOutRecordConsumer (later)
      • All the existing unit test will continue to pass, and be functionally equivalent (there may be minor compilation tweaks)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                danny.cranmer Danny Cranmer
                Reporter:
                danny.cranmer Danny Cranmer
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: