Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7004

Support configurable restore consumer poll timeout

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      In the StateChangelogReader, the restore consumer is currently hard coded to poll for 10ms at most per call.

       public Collection<TopicPartition> restore(final RestoringTasks active) {
          if (!needsInitializing.isEmpty()) {
              initialize();
          }
      
          if (needsRestoring.isEmpty()) {
              restoreConsumer.unsubscribe();
              return completed();
          }
      
          final Set<TopicPartition> restoringPartitions = new HashSet<>(needsRestoring.keySet());
          try {
              final ConsumerRecords<byte[], byte[]> allRecords = restoreConsumer.poll(10);
              for (final TopicPartition partition : restoringPartitions) {
                  restorePartition(allRecords, partition, active.restoringTaskFor(partition));
              }
          ...

       

      It'd be nice to be able to configure the restore consumer to poll for a larger timeout (e.g. 500ms) to give it more time to accumulate records for the restoration task. In the main event loop for polling in StreamThread.java, the main consumer uses the POLL_MS_CONFIG set in StreamsConfig.java to configure the max poll timeout. We could construct a similar config in the StreamsConfig class, but prefixed with the processing type (restore in this case). Let me know if this sounds reasonable, and I'll create a KIP and PR.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                shnguyen Shawn Nguyen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: