Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5487

In task processes, JobConf is unnecessarily loaded again in Limits

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: 2.3.0
    • Component/s: performance, task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time.

        Attachments

        1. MAPREDUCE-5487-1.patch
          10 kB
          Sandy Ryza
        2. MAPREDUCE-5487.patch
          5 kB
          Sandy Ryza

          Activity

            People

            • Assignee:
              sandyr Sandy Ryza
              Reporter:
              sandyr Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: