Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3537

DefaultContainerExecutor has a race condn. with multiple concurrent containers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: None
    • Labels:
      None

      Description

      DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3m 18s 1 Arun C Murthy 13/Dec/11 01:28
          Patch Available Patch Available Resolved Resolved
          5h 7m 1 Arun C Murthy 13/Dec/11 06:35
          Resolved Resolved Closed Closed
          82d 20h 13m 1 Arun C Murthy 05/Mar/12 02:48
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #926 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/)
          MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #926 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/ ) MAPREDUCE-3537 . Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #893 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/)
          MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #893 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/ ) MAPREDUCE-3537 . Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #124 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/124/)
          Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #124 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/124/ ) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #106 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/106/)
          Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #106 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/106/ ) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Daniel Dai made changes -
          Link This issue breaks PIG-2347 [ PIG-2347 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/286/)
          Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/286/ ) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1430 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1430/)
          MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1430 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1430/ ) MAPREDUCE-3537 . Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #274 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/274/)
          Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #274 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/274/ ) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1406 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1406/)
          MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1406 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1406/ ) MAPREDUCE-3537 . Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1480 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1480/)
          MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1480 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1480/ ) MAPREDUCE-3537 . Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213575 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #264 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/264/)
          Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #264 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/264/ ) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213576 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          Arun C Murthy made changes -
          Fix Version/s 0.23.1 [ 12318883 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Arun C Murthy added a comment -

          I just committed this.

          Show
          Arun C Murthy added a comment - I just committed this.
          Hide
          Mahadev konar added a comment -

          +1 the patch looks good.

          Show
          Mahadev konar added a comment - +1 the patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12507110/MAPREDUCE-3537.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507110/MAPREDUCE-3537.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//console This message is automatically generated.
          Arun C Murthy made changes -
          Link This issue is related to MAPREDUCE-3538 [ MAPREDUCE-3538 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.1 [ 12318883 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-3537.patch [ 12507110 ]
          Hide
          Arun C Murthy added a comment -

          Quick fix to add synchronization to unblock Pig, Oozie etc.

          We should redo the DCE more throughly.

          Show
          Arun C Murthy added a comment - Quick fix to add synchronization to unblock Pig, Oozie etc. We should redo the DCE more throughly.
          Arun C Murthy made changes -
          Field Original Value New Value
          Assignee Arun C Murthy [ acmurthy ]
          Description DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race.
          Arun C Murthy created issue -

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development