Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6391

output from ensure copartitioning is not used for Cluster metadata, resulting in partitions without tasks working on them

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394

      Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created from repartitionTopicMetadata
      THEN we do ensureCoPartitioning on repartitionTopicMetadata
      THEN we create topics and partitions according to repartitionTopicMetadata
      THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
      THEN we use fullMetadata to assign the tasks and no longer use repartitionTopicMetadata

      This results in any change to repartitionTopicMetadata in ensureCoPartitioning to be used for creating partitions but no tasks are ever created for any partition added by ensureCoPartitioning()

      the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata before creating allRepartitionTopicPartitions

        Attachments

          Activity

            People

            • Assignee:
              cvaliente Clemens Valiente
              Reporter:
              cvaliente Clemens Valiente
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 20m
                20m
                Remaining:
                Remaining Estimate - 20m
                20m
                Logged:
                Time Spent - Not Specified
                Not Specified