Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2115

Task cwds should be distributed across partitions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 0.14.3
    • None
    • None
    • None

    • All

    Description

      Even when mapred.local.dir specifies a comma-separated list of partitions (typically one per physical disk), all tasks of the same job have current working directories that belong to only one partition. For side-effect tasks, that use local cwd as a scratch space, this overloads a single disk while other disks may be idle. Idially, each task should get a cwd on different partition. This is related to HADOOP-1991, but emphasizes performance impact.

      Attachments

        Issue Links

          is part of

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-2227 wrong usage of mapred.local.dir.minspacestart

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              shv Konstantin Shvachko
              milindb Milind Barve
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: