Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9

mapred.local.dir temp dir. space allocation limited by smallest area

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

    • Hadoop Flags:
      Reviewed

      Description

      When mapred.local.dir is used to specify multiple temp dir. areas, space allocation limited by smallest area because the temp dir. selection algorithm is "round robin starting from a randomish point". When round robin is used with approximately constant sized chunks, the smallest area runs out of space first, and this is a fatal error.

      Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available areas.

      I wrote a patch to JobConf (currenly being tested) which uses df to check available space (once a minute or less often) and then uses an efficient roulette selection to do allocation weighted by magnitude of available space.

        Attachments

        1. hadoop9.patch
          7 kB
          Ariel Shemaiah Rabkin

          Activity

            People

            • Assignee:
              asrabkin Ariel Shemaiah Rabkin
              Reporter:
              pbaclace Paul Baclace
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: