Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5271

JobConf option for minimum progress threshold before reducers are assigned

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      A specific sub-case of the general priority inversion problem noted in HADOOP-4557 is when many lower priority jobs are submitted and are waiting for mappers to free up. Even though they haven't actually done any work, they will be assigned any free reducers. If a higher priority job is submitted, priority inversion results not just due to the lower priority tasks that are in the midst of completing, but also due to the ones that haven't yet started but have claimed all the free reducers. A simple workaround is to require a job to complete some useful work before assigning it a reducer. This can be done in a tunable and backwards compatible manner by adding a "minimum map progress percentage before assigning a reducer" option to the JobConf. Setting this to 0 would eliminate the common case above, and setting it to 100 would technically eliminate the inversion of HADOOP-4557, though likely at an unacceptably high cost.

      Attachments

        1. HADOOP-5271.patch
          6 kB
          Tim Williamson

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tim2 Tim Williamson
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: