Benchmarked the draft and v2 patches for producer throughput , here are the results -
Message size is 1K in all the tests
batch size 1, producer threads 1
kafka-736-v2 - 13 MB/s
kafka-736-draft - 30 MB/s
batch size 100, producer threads 1
kafka-736-v2 - 48.4 MB/s
kafka-736-draft - 61.5 MB/s
batch size 100, producer threads 20
kafka-736-v2 - 11.6 MB/s
kafka-736-draft - 81.6 MB/s
I looked into the cause of this performance degradation on the v2 patch. What's happening is setting the selection key's interest bits to READ in processNewResponses is not reflected in the following select() operation for all BUT the first network thread (id 0). I tried the producer performance test with varying # of producer threads and network threads on the server and I consistently see this result. Due to this, all the producer connections handled by network threads with ids > 1 see very low throughput since the next request is not read until 300 ms after the previous request is finished processing. I also confirmed that the producer had sent lot of data on those low throughput connections, just the server was reading it 300 ms later. I read up a little bit about concurrency and selection keys, found this -
"Generally, SelectionKey objects are thread-safe, but it's important to know that operations that modify the interest set are synchronized by Selector objects. This could cause calls to the interestOps( ) method to block for an indeterminate amount of time. The specific locking policy used by a selector, such as whether the locks are held throughout the selection process, is implementation-dependent.
Overall, seems like Java NIO doesn't behave the way we want to wrt to having the updated interest bits take effect in the next select operation. This makes the v2 approach even trickier to reason about.