Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20561

Use the position of the Kafka Consumer to track progress instead of Consumer Records offsets

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0-alpha-1
    • None
    • None

    Description

      Kafka Partitions with transactional messages (post 0.11) will include commit or abort markers which indicate the result of a transaction. The markers are not returned to applications, yet have an offset in the log. Therefore the end of Stream position can be the offset of a control message.
      This Patch change the way how we keep track of the consumer position by using

       consumer.position(topicP) 

      as oppose to using the offset of the consumed messages.
      Also I have done some refactoring to help code readability hopefully.

      Attachments

        1. HIVE-20561.3.patch
          18 kB
          Slim Bouguerra
        2. HIVE-20561.2.patch
          18 kB
          Slim Bouguerra
        3. HIVE-20561.patch
          18 kB
          Slim Bouguerra

        Issue Links

          Activity

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: