Thanks for the review ! I think I was over thinking the issue of request queue size having to be larger than io threads. Even if it is smaller, some io thread's shutdown will wait for some space to free up. Space will free up since some other io thread will dequeue the AllDone command.
This patch is very simple. It changed the shutdown logic of the Kafka server to go through following steps -
1. Shutdown acceptor, so no new connections are accepted
2. Shutdown processor threads, they will enqueue the currently selected keys' requests in the request queue. This is fine since io threads are alive and will dequeue requests. So this step will not block
3. Request channel shutdown will clear the queue. At this time, no thread is enqueuing more data. IO threads trying to dequeue data will hang on the receiveRequest
4. Shutdown io threads, this will enqueue AllDone command in the queue. And all io threads will shutdown one after the other. Even if the request queue is smaller than # of io threads, it will eventually shutdown