I started to work on this issue, but before any code, i would like to discuss a few things.
Current ipc/rpc handling is done as a 1 dispatcher N worker scenario. The Listener thread asynchronously listens to the socket for incoming connections, accepts them(creating Connection instances), or reads the data and puts in the queue for the handler threads for later processing.
The handler threads on the other hand, listens to the call queue, removes the first element and process the Call. It first makes the actual call to the method(through Server#call ), gets the return value and writes back the results to output stream(blocking).
As Doug suggested the performance might be improved if the results would have been written asynchronously. But i am not sure that it will affect the performance, if any, a lot. The costly operation in handler threads is expected to be the method calls. How do others think about this? We should start to think about developing benchmark tests. It would be really cool if we could improved the performance of ipc calls say %10. I am thinking about using jmeter for this, however maybe we need to develop our own benchmarking framework to test all the components of Hadoop. Any thoughts ?