Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-15228

camel-kafka - Missing end of polling signal in Processors

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.5.0
    • Component/s: camel-kafka
    • Labels:
      None
    • Estimated Complexity:
      Moderate

      Description

      For batch requirements it is currently not possible to get a signal for the last result within a poll request in processors (over all partitions and topics).

      The problem with this is, that a Processor cannot know when the last record of a poll received and it is time for commits to all of the affected partitions, because you cannot be sure that there is always a content in every partition and therefore you have no info in Processer, if any further message arrives.

      Solution could be to add a property to the manual commit handler in the KafkaConsumer.KafkaFetchRecords.doRun

       

      exchange.getIn().setHeader(KafkaConstants.MANUAL_COMMIT, manual);
      
      //adding new exchange property
      exchange.getIn().setHeader(KafkaConstants.LAST_RECORD_OF_CURRENT_POLL_REQUEST, last_record_of_last_partition);
      

      LAST_RECORD_IN_CURRENT_POLL_REQUEST should be a boolean to signal end of the loop over all partitions

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ramu11 Ramu
                Reporter:
                apjoe Jörg
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: