Kafka
  1. Kafka
  2. KAFKA-131

Hadoop Consumer goes into an infinite loop when kafka.request.limit is set to -1

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: contrib
    • Labels:

      Description

      There is a bug in KafkaETLContext.java where in a new Iterator instance is being created every time. This causes endless loops.

        Activity

        Hide
        Sam William added a comment -

        Adding an instance variable _respIterator , so that response.iterator() isn't called multiple times

        Show
        Sam William added a comment - Adding an instance variable _respIterator , so that response.iterator() isn't called multiple times
        Hide
        Sam William added a comment -

        Fix for the infinite loop bug in KafkaETLContext.java

        Show
        Sam William added a comment - Fix for the infinite loop bug in KafkaETLContext.java
        Hide
        Richard Park added a comment -

        Just one thing's missing.
        On line 201 (of original file):
        _offset += msgAndOffset.offset();

        That's incorrect. The msgAndOffset returns the offset, not the offset increment. So it should be:
        _offset = msgAndOffset.offset();

        Show
        Richard Park added a comment - Just one thing's missing. On line 201 (of original file): _offset += msgAndOffset.offset(); That's incorrect. The msgAndOffset returns the offset, not the offset increment. So it should be: _offset = msgAndOffset.offset();
        Hide
        Sam William added a comment -

        Adding the fix for offsets (Line 201)

        Show
        Sam William added a comment - Adding the fix for offsets (Line 201)
        Hide
        Richard Park added a comment -

        Great. I'm good with this patch.

        Show
        Richard Park added a comment - Great. I'm good with this patch.
        Hide
        Jun Rao added a comment -

        Thanks Sam and Richard. I just committed this.

        Show
        Jun Rao added a comment - Thanks Sam and Richard. I just committed this.
        Hide
        Felix GV added a comment -

        I just wanted to point out that this bug seems to happen whether kafka.request.limit is set to -1 or not.

        The current 0.6 release that is available on the site is not very usable because of this bug...

        The current trunk does fix this problem though, which is great. Thanks !

        Show
        Felix GV added a comment - I just wanted to point out that this bug seems to happen whether kafka.request.limit is set to -1 or not. The current 0.6 release that is available on the site is not very usable because of this bug... The current trunk does fix this problem though, which is great. Thanks !
        Hide
        Blake Matheny added a comment -

        We have run into this as well Felix. I'd like to backport whatever change fixed this in trunk into our 0.6.1 branch. Any idea where I should look?

        Show
        Blake Matheny added a comment - We have run into this as well Felix. I'd like to backport whatever change fixed this in trunk into our 0.6.1 branch. Any idea where I should look?
        Hide
        Sam William added a comment -

        Blake,
        You could apply the attached patch (https://issues.apache.org/jira/secure/attachment/12493181/KAFKA-131.patch) to the file KafkaETLContext.java

        Show
        Sam William added a comment - Blake, You could apply the attached patch ( https://issues.apache.org/jira/secure/attachment/12493181/KAFKA-131.patch ) to the file KafkaETLContext.java
        Hide
        Blake Matheny added a comment -

        Sorry I should have been more clear. We actually run into this issue not using the KafkaETL. We occasionally see regular consumers go into a loop (continue to fetch the same offset), I thought the comment from Felix was referring to that specifically.

        Show
        Blake Matheny added a comment - Sorry I should have been more clear. We actually run into this issue not using the KafkaETL. We occasionally see regular consumers go into a loop (continue to fetch the same offset), I thought the comment from Felix was referring to that specifically.

          People

          • Assignee:
            Unassigned
            Reporter:
            Sam William
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development