Would be nice to do the following to aid debugging and help solve latency issues:
- Add some basic Trace(const string& annotation) call to RpcContext
- This would store the log message in a RPC-local buffer, perhaps with some basic sampling ability
- Collect latency histograms for each RPC call on the server, and a reservoir sample for each bucket of the histogram. The sampled traces would include the request, response, and trace
- Expose the above info through an /rpcz interface
Would be cool to work with the HBase folks on this too, so we end up building something that exposes the same data model and CM can pick up the info.