Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3962

Update ConsumeKafkaRecord_0_10 to better batch together messages into fewer flowfiles

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0
    • Extensions
    • None

    Description

      Currently, ConsumeKafkaRecord will batch together only as many records in a FlowFile as it is able to obtain for a single partition & topic. This should be updated to allow for multiple iterations of KafkaConsumer.poll() in the same way that ConsumeKafka does with a Demarcator configured.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. NIFI-7642 KafkaConsumers: Improve batching into fewer FlowFiles under high latency conditions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #1846

          Activity

            People

              markap14 Mark Payne
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: