Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3280

KafkaConsumer Javadoc contains misleading description of heartbeat behavior and correct use

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 0.10.0.0
    • Component/s: consumer
    • Labels:
    • Flags:
      Patch

      Description

      The KafkaConsumer Javadoc says that: "The consumer will automatically ping the cluster periodically, which lets the cluster know that it is alive. As long as the consumer is able to do this it is considered alive and retains the right to consume from the partitions assigned to it." This is false. The heartbeat process is neither automatic nor periodic. The consumer heartbeats exactly once when poll() is called. The consumer's run thread is responsible for calling poll() before session.timeout.ms elapses.

      Based on this misinformation, it is easy for a naive implementer to build a batch-based kafka consumer that takes longer than session.timeout.ms between poll() calls and encounter very ugly rebalance loops that can be very hard to diagnose. Clarification in the docs would help a lot--I'll submit a patch shortly.

        Attachments

          Activity

            People

            • Assignee:
              nehanarkhede Neha Narkhede
              Reporter:
              rwhaling Richard Whaling
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 0.25h
                0.25h
                Remaining:
                Remaining Estimate - 0.25h
                0.25h
                Logged:
                Time Spent - Not Specified
                Not Specified