Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27773

Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 3.0.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      The health of the external shuffle service is currently difficult to monitor. At least for the YARN shuffle service, the only current indication of health is whether or not the shuffle service threads are running in the NodeManager. However, we've seen that clients can sometimes experience elevated failure rates on requests to the shuffle service even when those threads are running. It would be helpful to have some indication of how often requests to the shuffle service are failing, as this could be monitored, alerted on, etc.

      One suggestion (implemented in the PR I'll attach to this ticket) is to add a metric to ExternalShuffleBlockHandler.ShuffleMetrics which keeps track of how many times we caught an exception in the shuffle service's RPC handler. I think that this gives us the insight into request failure rates that we're currently missing, but obviously I'm open to alternatives as well if people have other ideas.

        Attachments

          Activity

            People

            • Assignee:
              Steven Rand Steven Rand
              Reporter:
              Steven Rand Steven Rand
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: