Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25642

Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Shuffle, Spark Core
    • None

    Description

      Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364|https://github.com/apache/spark/pull/22485]. We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments.

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-31646 Remove unused registeredConnections counter from ShuffleMetrics

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #22498 [Github] Pull Request #22498 (pgandhi999)

          Pull request #22498 [Github] Pull Request #22498 (pgandhi999)

          Web Link GitHub Pull Request #22498

          Activity

            People

              pgandhi Parth Gandhi
              pgandhi Parth Gandhi
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: