Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7794

kafka.tools.GetOffsetShell does not return the offset in some cases

    XMLWordPrintableJSON

Details

    Description

      For some input for the timestamps (different from -1 or -2) the GetOffset is not able to retrieve the offset.

      For example, if x is the timestamp in that "not working range", and you execute:

      bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_ADDRESS --topic $MY_TOPIC --time x
      

      The output is:

      MY_TOPIC:8:
      MY_TOPIC:2:
      MY_TOPIC:5:
      MY_TOPIC:4:
      MY_TOPIC:7:
      MY_TOPIC:1:
      MY_TOPIC:9:

      while after the last ":" an integer representing the offset is expected.


      Steps to reproduce it:

      1. Consume all the messages from the beginning and print the timestamp:
        bin/kafka-simple-consumer-shell.sh --no-wait-at-logend --broker-list $KAFKA_ADDRESS --topic $MY_TOPIC --property print.timestamp=true  > messages
      1. Sort the messages by timestamp and get some of the oldest messages:
         awk -F "CreateTime:" '{ print $2}' messages | sort -n > msg_sorted
      1. Take (for example) the timestamp of the 10th oldest message, and see if GetOffsetShell is not able to print the offset:
        timestamp="$(sed '10q;d' msg_sorted | cut -f1)"
        bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_ADDRESS --topic $MY_TOPIC --time $timestamp
        # The output should be something like:
        # MY_TOPIC:1:
        # MY_TOPIC:2:
        (repeated for every partition)
      1. Verify that the message with that timestamp is still in Kafka:
        bin/kafka-simple-consumer-shell.sh --no-wait-at-logend --broker-list $KAFKA_ADDRESS --topic $MY_TOPIC --property print.timestamp=true | grep "CreateTime:$timestamp" 

       

      Attachments

        1. image-2019-02-16-22-24-11-799.png
          19 kB
          Kartik
        2. image-2019-02-13-11-45-21-459.png
          56 kB
          Kartik
        3. image-2019-02-13-11-44-18-736.png
          63 kB
          Kartik
        4. image-2019-02-13-11-43-28-873.png
          17 kB
          Kartik
        5. image-2019-02-13-11-43-24-128.png
          17 kB
          Kartik
        6. image-2019-02-12-16-23-38-399.png
          21 kB
          Daniele Ascione
        7. image-2019-02-12-16-21-13-126.png
          33 kB
          Daniele Ascione
        8. image-2019-02-12-16-19-25-170.png
          28 kB
          Daniele Ascione
        9. image-2019-02-11-20-57-03-579.png
          16 kB
          Kartik
        10. image-2019-02-11-20-56-13-362.png
          25 kB
          Kartik
        11. image-2019-02-11-20-51-07-805.png
          23 kB
          Kartik

        Activity

          People

            kartikvk1996 Kartik
            audhumla Daniele Ascione
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: