Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5736

Jobtracker to hang when jobs with lot of tasks running

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: jobtracker
    • Labels:
      None

      Description

      The jobtracker (in MRv1) is progressing slowly when a job with lot of tasks is running. The reason is that JT.getJobCounters is holding a global lock, and with a big job (like 50+K mappers for instance), it could take while to instanciate the ``Counters`` class. This global lock prevent all other activities to run normally, queuing them and degrading the normal activities of the JT.

      I was wondering if job.getCounters(), which is synchronized on a finer granularity (i.e. per job and not global) couldn't be taken out of the synchronized block.

      On a more general idea, I was wondering if the usage of the synchronized statement in the JT shouldn't be re-thought. Or maybe all this has already been addressed in YARN.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bperroud Benoit Perroud
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: