Minor Description
A TransactionManager instance is created by the KafkaProducer and shared with the Sender thread. The TransactionManager has internal states through which it transitions as part of its initialization, transaction management, shutdown, etc. It contains logic to ensure that those state transitions are valid, such that when an invalid transition is attempted, it is handled appropriately.
The issue is, the definition of "handled appropriately" depends on which thread is making the API call that is attempting an invalid transition. The application thread expects that the invalid transition will generate an exception. However, the sender thread expects that the invalid transition will "poison" the entire TransactionManager instance.
So as part of the implementation of KAFKA-14831, we needed a way to change logic in the TransactionManager on a per-thread basis, so a ThreadLocal instance variable was added to the TransactionManager to keep track of this. However, the use of ThreadLocal instance variables is generally discouraged because of their tendency for memory leaks, shared state across multiple threads, and not working with virtual threads.
The initial implementation attempt of KAFKA-14831 used a context object, passed in to each method, to affect the logic. However, the number of methods that needed to be changed to accept this new context object grew until most of the methods in TransactionManager needed to be updated. Thus all the affected call sites needed to be updated, resulting in a much larger change than anticipated.
The task here is to remove the use of the ThreadLocal instance variable, let the application thread and Sender thread keep their desired behavior, but keep the change to a minimum.
Attachments
Issue Links
- is caused by
-
KAFKA-14831 Illegal state errors should be fatal in transactional producer
-
- Resolved
-