Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7070

Batch calls to RackResolver from AM

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • applicationmaster, performance
    • None
    • Patch

    Description

      TaskAttemptImpl calls RackResolver once for every split location. This results in many shells being spawned by the Application Master when the topology script is enabled. The worst case scenario would entail one call to the topology script for each Datanode. The NameNode is also topology aware, and uses batch calls to the topology script to improve performance.

      Attachments

        1. MAPREDUCE-7070.patch
          5 kB
          Greg Phillips

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-8170 Caching Node Rack Location

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              gphillips Greg Phillips
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: