Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10806

Support multiple consuming offsets when discovering a new topic

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2, 1.8.1
    • Fix Version/s: None
    • Component/s: Connectors / Kafka
    • Labels:
      None

      Description

      In KafkaConsumerBase, we discover the TopicPartitions and compare them with the restoredState. It's reasonable when a topic's partitions scaled. However, if we add a new topic which has too much data and restore the Flink program, the data of the new topic will be consumed from the start, which may not be what we want. I think this should be an option for developers.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wind_ljy Jiayi Liao
                Reporter:
                wind_ljy Jiayi Liao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: