Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3988

mapreduce.job.local.dir doesn't point to a single directory on a node.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node.

      It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs.

      1. MAPREDUCE-3988-2.txt
        4 kB
        Eric Payne
      2. MAPREDUCE-3988-2.txt
        4 kB
        Eric Payne
      3. MAPREDUCE-3988-1.txt
        4 kB
        Eric Payne

        Activity

        Vinod Kumar Vavilapalli created issue -
        Eric Payne made changes -
        Field Original Value New Value
        Assignee Eric Payne [ eepayne ]
        Hide
        Eric Payne added a comment -

        This patch uses the LocalDirAllocator.getLocalPathToRead() API to first check for existence of a scratch directory before using the LocalDirAllocator.getLocalPathForWrite() API if the directory does not exist.

        There are a couple of race conditions with this implementation because of the fact that each YarnChild task attempt is running in its own JVM. This patch addresses one of the race conditions.

        The race addressed by this patch is when 2 or more tasks at the same time detect there is no mapreduce.job.local.dir and try to create it in the same directory at the same time. Whichever one loses the race will get a FileAlreadyExistsException, which is ignored.

        There is still a race condition wherein 2 tasks running on the same node could both see that there is no mapreduce.job.local.dir and then try to create it in 2 different directories; for eg, one in /grid/0 and one in /grid/1.

        We could go a step further in addressing the second race and try to hash the attempt ID and map it to the same directory, but that seems overly complicated.

        Show
        Eric Payne added a comment - This patch uses the LocalDirAllocator.getLocalPathToRead() API to first check for existence of a scratch directory before using the LocalDirAllocator.getLocalPathForWrite() API if the directory does not exist. There are a couple of race conditions with this implementation because of the fact that each YarnChild task attempt is running in its own JVM. This patch addresses one of the race conditions. The race addressed by this patch is when 2 or more tasks at the same time detect there is no mapreduce.job.local.dir and try to create it in the same directory at the same time. Whichever one loses the race will get a FileAlreadyExistsException, which is ignored. There is still a race condition wherein 2 tasks running on the same node could both see that there is no mapreduce.job.local.dir and then try to create it in 2 different directories; for eg, one in /grid/0 and one in /grid/1. We could go a step further in addressing the second race and try to hash the attempt ID and map it to the same directory, but that seems overly complicated.
        Eric Payne made changes -
        Attachment MAPREDUCE-3988-1.txt [ 12518139 ]
        Eric Payne made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518139/MAPREDUCE-3988-1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518139/MAPREDUCE-3988-1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2042//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Looking at the patch I think it looks good to me, but I would like Vinod's opinion on it before checking it in, because the last time I jumped the gun and that is why we need this patch.

        Show
        Robert Joseph Evans added a comment - Looking at the patch I think it looks good to me, but I would like Vinod's opinion on it before checking it in, because the last time I jumped the gun and that is why we need this patch.
        Eric Payne made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Eric Payne added a comment -

        Upmerged patch to branch-0.23, branch-2 and trunk.

        Same patch applies to all.

        Show
        Eric Payne added a comment - Upmerged patch to branch-0.23, branch-2 and trunk. Same patch applies to all.
        Eric Payne made changes -
        Attachment MAPREDUCE-3988-2.txt [ 12520985 ]
        Eric Payne made changes -
        Attachment MAPREDUCE-3988-2.txt [ 12520987 ]
        Eric Payne made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12520987/MAPREDUCE-3988-2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520987/MAPREDUCE-3988-2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2126//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Eric. I checked this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Eric. I checked this into trunk, branch-2, and branch-0.23
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Fix Version/s 2.0.0 [ 12320354 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2057/)
        MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2057/ ) MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1982 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1982/)
        MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1982 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1982/ ) MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1995 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1995/)
        MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1995 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1995/ ) MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #218 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/)
        svn merge -c 1309086 from trunk. FIXES MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309088)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309088
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #218 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/ ) svn merge -c 1309086 from trunk. FIXES MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309088) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309088 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/)
        MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/ ) MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/)
        MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/ ) MAPREDUCE-3988 . mapreduce.job.local.dir doesn't point to a single directory on a node. (Eric Payne via bobby) (Revision 1309086) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309086 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 2.0.0-alpha [ 12320354 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        20d 14h 59m 1 Eric Payne 02/Apr/12 17:45
        Open Open Patch Available Patch Available
        4d 6h 6m 2 Eric Payne 02/Apr/12 17:51
        Patch Available Patch Available Resolved Resolved
        1d 1h 40m 1 Robert Joseph Evans 03/Apr/12 19:31
        Resolved Resolved Closed Closed
        50d 1h 56m 1 Arun C Murthy 23/May/12 21:28

          People

          • Assignee:
            Eric Payne
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development