Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12809

PublishKafkaRecord_2_6 - RoundRobin partitioner skipping every other partition

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      When configured to use the NiFi RoundRobin partitioner, the processor publishes to every other partition. If the number of partitions in the topic and the number of records being published are the right combination, this problem is masked. We see this issue when we set the partitions to 26, but not when set to 25. 

      I took a code-dive into the o.a.k.c.producer.KafkaProducer and discovered that it is invoking the Partitioner twice when a "new batch" is created. Thus, the RoundRobin partitioner bumps the index by 2. If the RoundRobin partitioner overwrote the onNewBatch method, this problem could be solved.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            slyouts Steven Youtsey

            Dates

              Created:
              Updated:

              Slack

                Issue deployment