Interesting stuff, guys. many good points have been brought up.
The code path for UNIX domain sockets is like this now:
1. server calls accept(), gets a socket, hands it off to worker thread
2. worker thread reads the RPC header to find the type of message and length
3. worker thread reads the message
4. worker thread processes the message
Having QoS information in the header would allow us to prioritize the message after step #2.
Having QoS information in the protobuf would allow us to prioritize the message after step #3.
Since messages are normally just a few bytes, I'm not sure that this would be a big win.
In general, I think using a separate UNIX domain socket would probably make more sense. It would also allow us to use operating system features like the accept backlog to our advantage-- when using a single socket, we have to implement all that ourselves, and we don't really have the tools in userspace to do a good job.