Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2289

Permissions race can make getStagingDir fail on local filesystem

    Details

    • Hadoop Flags:
      Reviewed

      Description

      I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem:

      • two threads call getStagingDir at the same time
      • Thread A checks fs.exists(stagingArea) and sees false
        • Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS)
          • mkdirs calls the Java mkdir API which makes the file with umask-based permissions
      • Thread B runs, checks fs.exists(stagingArea) and sees true
        • checks permissions, sees the default permissions, and throws IOE
      • Thread A resumes and sets correct permissions
      1. MAPREDUCE-2289_branch-1.0.patch
        3 kB
        Ahmed Radwan
      2. MAPREDUCE-2289_trunk.patch
        3 kB
        Ahmed Radwan
      3. mapreduce-2289.txt
        3 kB
        Todd Lipcon

        Issue Links

          Activity

          Todd Lipcon created issue -
          Hide
          Todd Lipcon added a comment -

          The central issue here is that the Java mkdir API doesn't take a mode, so whenever we create a directory, it starts out with umask-based permissions until we can call chmod on it.

          Seems to me it would be safe and fix the race if we had it fix permissions if they were wrong, rather than bailing out. Then the race would only cause a redundant chmod. Thoughts?

          Show
          Todd Lipcon added a comment - The central issue here is that the Java mkdir API doesn't take a mode, so whenever we create a directory, it starts out with umask-based permissions until we can call chmod on it. Seems to me it would be safe and fix the race if we had it fix permissions if they were wrong, rather than bailing out. Then the race would only cause a redundant chmod. Thoughts?
          Hide
          Todd Lipcon added a comment -

          Here's a patch which implements the proposed change (fixing submission dir permissions if they're incorrect)

          Show
          Todd Lipcon added a comment - Here's a patch which implements the proposed change (fixing submission dir permissions if they're incorrect)
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment mapreduce-2289.txt [ 12469984 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12469984/mapreduce-2289.txt
          against trunk revision 1074251.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469984/mapreduce-2289.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/34//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.

          Show
          Arun C Murthy added a comment - Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Patrick Hunt made changes -
          Link This issue breaks MAPREDUCE-2976 [ MAPREDUCE-2976 ]
          Konstantin Shvachko made changes -
          Fix Version/s 0.22.1 [ 12319242 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Hide
          Ahmed Radwan added a comment -

          Here are updated patched that apply cleanly to both trunk and branch-1.0.

          Show
          Ahmed Radwan added a comment - Here are updated patched that apply cleanly to both trunk and branch-1.0.
          Ahmed Radwan made changes -
          Attachment MAPREDUCE-2289_branch-1.0.patch [ 12531711 ]
          Attachment MAPREDUCE-2289_trunk.patch [ 12531712 ]
          Ahmed Radwan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531712/MAPREDUCE-2289_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531712/MAPREDUCE-2289_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2449//console This message is automatically generated.
          Ahmed Radwan made changes -
          Assignee Todd Lipcon [ tlipcon ] Ahmed Radwan [ ahmed.radwan ]
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Ahmed. Committed to trunk, branch-1 & branch-2.

          Show
          Alejandro Abdelnur added a comment - Thanks Ahmed. Committed to trunk, branch-1 & branch-2.
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 2.0.1-alpha [ 12321442 ]
          Fix Version/s 0.22.1 [ 12319242 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2386 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2386/)
          MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2386 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2386/ ) MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2456 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2456/)
          MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2456 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2456/ ) MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2404 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2404/)
          MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2404 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2404/ ) MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/)
          MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/ ) MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/)
          MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/ ) MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1353750) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353750 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Robert Joseph Evans made changes -
          Fix Version/s 0.23.3 [ 12320060 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #305 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/305/)
          svn merge -c 1353750 FIXES: MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1358400)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358400
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #305 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/305/ ) svn merge -c 1353750 FIXES: MAPREDUCE-2289 . Permissions race can make getStagingDir fail on local filesystem (ahmed via tucu) (Revision 1358400) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358400 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322471 ]
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 2.1.0-alpha [ 12321442 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          added 1.2.0 to fixVersion, per CHANGES.txt and svn logs.

          Show
          Matt Foley added a comment - added 1.2.0 to fixVersion, per CHANGES.txt and svn logs.
          Matt Foley made changes -
          Fix Version/s 1.2.0 [ 12321661 ]

            People

            • Assignee:
              Ahmed Radwan
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development