Major Description
Since you actually can't run multiple instances of a Kafka Streams application on the same physical state directory, there's really no benefit to the file locks we obtain as part of the task directory locking. The only safety measures we need are to protect between threads within a process, not across processes. The in-memory map of locks should be sufficient for the StateDirectory to be safe.
So, we should take out all of the file-based locking of task directories. This should allow us to greatly simplify the StateDirectory code, and eliminate the source of some problems we've faced in the past, particularly those finicky FS/OS dependent issues
Attachments
Issue Links
- links to