Details
Description
The ConsumerRebalanceListener used by the framework to respond to rebalance events in consumer groups for sink tasks is hard-coded with the assumption that the consumer performs rebalances eagerly. In other words, it assumes that whenever onPartitionsRevoked is called, all partitions have been revoked from that consumer, and whenever onPartitionsAssigned is called, the partitions passed in to that method comprise the complete set of topic partitions assigned to that consumer.
See the WorkerSinkTask.HandleRebalance class for the specifics.
One issue this can cause is silently ignoring to-be-committed offsets provided by sink tasks, since the framework ignores offsets provided by tasks in their preCommit method if it does not believe that the consumer for that task is currently assigned the topic partition for that offset. See these lines in the WorkerSinkTask::commitOffsets method for reference.
This may not be the only issue caused by configuring a sink connector's consumer to use cooperative rebalancing. Rigorous unit and integration testing should be added before claiming that the Connect framework supports the use of cooperative consumers with sink connectors.
Attachments
Issue Links
- causes
-
KAFKA-13472 Connect can lose track of last committed offsets for topic partitions after partial consumer revocation
-
- Resolved
-
- is duplicated by
-
-
- Resolved
-
- is related to
-
KAFKA-12463 Update default consumer partition assignor for sink tasks
-
- Open
-
-
-
- Resolved
-
- links to