Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1529

Add Localization overhead metrics to NM

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.2, 2.10.1, 3.4.0, 3.3.1
    • nodemanager
    • None

    Description

      Users are often unaware of localization cost that their jobs incur. To measure effectiveness of localization caches it is necessary to expose the overhead in the form of metrics.

      We propose addition of the following metrics to NodeManagerMetrics.

      When a container is about to launch, its set of LocalResources has to be fetched from a central location, typically on HDFS, that results in a number of download requests for the files missing in caches.

      LocalizedFilesMissed: total files (requests) downloaded from DFS. Cache misses.

      LocalizedFilesCached: total localization requests that were served from local caches. Cache hits.

      LocalizedBytesMissed: total bytes downloaded from DFS due to cache misses.

      LocalizedBytesCached: total bytes satisfied from local caches.

      Localized(Files|Bytes)CachedRatio: percentage of localized (files|bytes) that were served out of cache: ratio = 100 * caches / (caches + misses)

      LocalizationDownloadNanos: total elapsed time in nanoseconds for a container to go from ResourceRequestTransition to LocalizedTransition

      Attachments

        1. YARN-1529.v01.patch
          14 kB
          Gera Shegalov
        2. YARN-1529.v02.patch
          21 kB
          Gera Shegalov
        3. YARN-1529.v03.patch
          20 kB
          Gera Shegalov
        4. YARN-1529.v04.patch
          20 kB
          Jason Darrell Lowe
        5. YARN-1529.005.patch
          21 kB
          Jim Brennan
        6. YARN-1529.006.patch
          22 kB
          Jim Brennan
        7. YARN-1529-branch-2.10.001.patch
          22 kB
          Jim Brennan

        Issue Links

          Activity

            People

              jbrennan Jim Brennan
              jira.shegalov Gera Shegalov
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: