Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.4.3
-
None
Description
The health of the external shuffle service is currently difficult to monitor. At least for the YARN shuffle service, the only current indication of health is whether or not the shuffle service threads are running in the NodeManager. However, we've seen that clients can sometimes experience elevated failure rates on requests to the shuffle service even when those threads are running. It would be helpful to have some indication of how often requests to the shuffle service are failing, as this could be monitored, alerted on, etc.
One suggestion (implemented in the PR I'll attach to this ticket) is to add a metric to ExternalShuffleBlockHandler.ShuffleMetrics which keeps track of how many times we caught an exception in the shuffle service's RPC handler. I think that this gives us the insight into request failure rates that we're currently missing, but obviously I'm open to alternatives as well if people have other ideas.