Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2725

high level consumer rebalances with auto-commit disabled should throw an exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Auto Closed
    • 0.8.2.1
    • None
    • clients
    • None
    • Experienced on Java running in linux

    Description

      Auto-commit is a very resilient mode. Drops in zookeeper sessions due to JVM garbage collection, network, rebalance or other interference are handled gracefully within the kafka client.

      Systems still can drop due to unexpected gc or network behavior. My proposal is to handle this drop better when auto-commit is turned off:

      • If a rebalance or similar occur (which cause the offset to get reverted in the client), check and see if the client was assigned back to the same partition or a different one. If it's the same partition, find the place last consumed (it doesn't do this today for us). This is to make a graceful recovery.
      • If the partition assignment changes (which can mean duplicate data is getting processed), throw an exception back to the application code. This lets the application code handle this exception-case with respect to the work it's doing (with might be transactional). Failing "silently" (yes it's still getting logged) is very dangerous in our situation.

      Attachments

        Activity

          People

            Unassigned Unassigned
            crhyne Cliff Rhyne
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: