Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11347

RawLocalFileSystem#mkdir and create should honor umask

    Details

    • Hadoop Flags:
      Incompatible change

      Description

      The FileSystem and FileContext APIs are inconsistent in enforcement of umask for newly created directories. FileContext utilizes configuration property fs.permissions.umask-mode and runs a separate chmod call to guarantee bypassing the process umask. This is the expected behavior for Hadoop as discussed in the documentation of fs.permissions.umask-mode. For the equivalent FileSystem APIs, it does not use fs.permissions.umask-mode. Instead, the permissions end up getting controlled by the process umask.

      1. HADOOP-11347.001.patch
        14 kB
        Varun Saxena
      2. HADOOP-11347.002.patch
        18 kB
        Varun Saxena
      3. HADOOP-11347.03.patch
        15 kB
        Varun Saxena
      4. HADOOP-11347.004.patch
        9 kB
        Varun Saxena
      5. HADOOP-11347.05.patch
        9 kB
        Varun Saxena

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          More details are in this comment from HADOOP-11321:

          https://issues.apache.org/jira/browse/HADOOP-11321?focusedCommentId=14230778&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14230778

          Note that this issue is targeted to 3.0.0. I don't believe we can change the behavior safely on the 2.x line. Applications may be relying on the existing behavior of FileSystem.

          Show
          cnauroth Chris Nauroth added a comment - More details are in this comment from HADOOP-11321 : https://issues.apache.org/jira/browse/HADOOP-11321?focusedCommentId=14230778&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14230778 Note that this issue is targeted to 3.0.0. I don't believe we can change the behavior safely on the 2.x line. Applications may be relying on the existing behavior of FileSystem .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12687133/HADOOP-11347.001.patch
          against trunk revision 25a0440.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.crypto.key.TestKeyProviderFactory
          org.apache.hadoop.crypto.random.TestOsSecureRandom
          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.security.alias.TestCredentialProviderFactory
          org.apache.hadoop.fs.TestHarFileSystem

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687133/HADOOP-11347.001.patch against trunk revision 25a0440. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.crypto.key.TestKeyProviderFactory org.apache.hadoop.crypto.random.TestOsSecureRandom org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.security.alias.TestCredentialProviderFactory org.apache.hadoop.fs.TestHarFileSystem Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5265//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12687146/HADOOP-11347.002.patch
          against trunk revision 25a0440.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.crypto.random.TestOsSecureRandom
          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687146/HADOOP-11347.002.patch against trunk revision 25a0440. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.crypto.random.TestOsSecureRandom org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5267//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Findbugs and test failures unrelated.

          Show
          varun_saxena Varun Saxena added a comment - Findbugs and test failures unrelated.
          Hide
          varun_saxena Varun Saxena added a comment -
          Show
          varun_saxena Varun Saxena added a comment - Chris Nauroth , Colin P. McCabe , kindly review
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12687146/HADOOP-11347.002.patch
          against trunk revision ca1c00b.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5812//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687146/HADOOP-11347.002.patch against trunk revision ca1c00b. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5812//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Varun Saxena, are going to do a revision of this one? I still think we should do this in 3.0 as an incompatible change.

          Show
          cmccabe Colin P. McCabe added a comment - Varun Saxena , are going to do a revision of this one? I still think we should do this in 3.0 as an incompatible change.
          Hide
          varun_saxena Varun Saxena added a comment -

          Colin P. McCabe, updated a new patch after rebase.

          Show
          varun_saxena Varun Saxena added a comment - Colin P. McCabe , updated a new patch after rebase.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 29s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 1 new checkstyle issues (total was 163, now 163).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 22m 22s Tests passed in hadoop-common.
              59m 8s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733404/HADOOP-11347.03.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cab0dad
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 1 new checkstyle issues (total was 163, now 163). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 22s Tests passed in hadoop-common.     59m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733404/HADOOP-11347.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cab0dad checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6722/console This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Checkstyle issue is related to file length which is not under the scope of this JIRA.

          Show
          varun_saxena Varun Saxena added a comment - Checkstyle issue is related to file length which is not under the scope of this JIRA.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for looking at this, Varun. I don't think we need to change the FileSystem base class. This JIRA is about the local file system-- that's the FS that is having trouble with this, and that's the one that should change.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for looking at this, Varun. I don't think we need to change the FileSystem base class. This JIRA is about the local file system-- that's the FS that is having trouble with this, and that's the one that should change.
          Hide
          varun_saxena Varun Saxena added a comment -

          Ok...Will make the necessary change.

          Show
          varun_saxena Varun Saxena added a comment - Ok...Will make the necessary change.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 1 new checkstyle issues (total was 22, now 23).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 33s Tests passed in hadoop-common.
              61m 39s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736428/HADOOP-11347.04.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / da05361
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 1 new checkstyle issues (total was 22, now 23). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 33s Tests passed in hadoop-common.     61m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736428/HADOOP-11347.04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / da05361 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6882/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 6s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 23s Tests passed in hadoop-common.
              61m 19s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736452/HADOOP-11347.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c2356cb
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 6s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 23s Tests passed in hadoop-common.     61m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736452/HADOOP-11347.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c2356cb hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6883/console This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Colin P. McCabe, kindly review

          Show
          varun_saxena Varun Saxena added a comment - Colin P. McCabe , kindly review
          Hide
          cmccabe Colin P. McCabe added a comment -

          I'll try to take a look later today. Thanks for your patience.

          Show
          cmccabe Colin P. McCabe added a comment - I'll try to take a look later today. Thanks for your patience.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for looking at this, Varun Saxena.

          private FsPermission umask = new FsPermission((short)0022);

          I don't think this needs to have a default value, given that initialize should always be called first. It can just be null until the FS is initialized.

          @@ -296,12 +299,16 @@ private FSDataOutputStream create(Path f, boolean overwrite,
             
             protected OutputStream createOutputStream(Path f, boolean append) 
                 throws IOException {
          -    return createOutputStreamWithMode(f, append, null);
          +    return createOutputStreamWithMode(f, append, FsPermission.getFileDefault());
             }
           
             protected OutputStream createOutputStreamWithMode(Path f, boolean append,
                 FsPermission permission) throws IOException {
          -    return new LocalFSFileOutputStream(f, append, permission);
          +    if (null == permission) {
          +      permission = FsPermission.getFileDefault();
          +    }
          +    return new LocalFSFileOutputStream(
          +        f, append, permission.applyUMask(umask));
             }
          

          Rather than changing this, I would prefer to change the LocalFSFileOutputStream to apply the umask to the permission that was passed in. It's a package-private constructor only used in this file, so we should be able to change it without compatibility issues. That way we would have fewer pieces of code that needed to be dealing with umask. Similarly with mkdir / mkOneDirWithMode.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for looking at this, Varun Saxena . private FsPermission umask = new FsPermission((short)0022); I don't think this needs to have a default value, given that initialize should always be called first. It can just be null until the FS is initialized. @@ -296,12 +299,16 @@ private FSDataOutputStream create(Path f, boolean overwrite, protected OutputStream createOutputStream(Path f, boolean append) throws IOException { - return createOutputStreamWithMode(f, append, null ); + return createOutputStreamWithMode(f, append, FsPermission.getFileDefault()); } protected OutputStream createOutputStreamWithMode(Path f, boolean append, FsPermission permission) throws IOException { - return new LocalFSFileOutputStream(f, append, permission); + if ( null == permission) { + permission = FsPermission.getFileDefault(); + } + return new LocalFSFileOutputStream( + f, append, permission.applyUMask(umask)); } Rather than changing this, I would prefer to change the LocalFSFileOutputStream to apply the umask to the permission that was passed in. It's a package-private constructor only used in this file, so we should be able to change it without compatibility issues. That way we would have fewer pieces of code that needed to be dealing with umask. Similarly with mkdir / mkOneDirWithMode.
          Hide
          varun_saxena Varun Saxena added a comment -

          Colin P. McCabe, will make the necessary change.

          Show
          varun_saxena Varun Saxena added a comment - Colin P. McCabe , will make the necessary change.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 6s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 13s Tests passed in hadoop-common.
              62m 7s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738259/HADOOP-11347.05.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b61b489
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 6s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 13s Tests passed in hadoop-common.     62m 7s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738259/HADOOP-11347.05.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b61b489 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6935/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thanks, Varun Saxena.

          I will commit this to trunk. Since this is incompatible, we can talk about whether to commit to branch-2 later.

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thanks, Varun Saxena . I will commit this to trunk. Since this is incompatible, we can talk about whether to commit to branch-2 later.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7994/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7994/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/223/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/223/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #953 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/953/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #953 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/953/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks for the review and commit Colin P. McCabe

          Show
          varun_saxena Varun Saxena added a comment - Thanks for the review and commit Colin P. McCabe
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2151 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2151/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2151 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2151/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/212/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/212/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/221/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/221/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2169 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2169/)
          HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2169 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2169/ ) HADOOP-11347 . RawLocalFileSystem#mkdir and create should honor umask (Varun Saxena via Colin P. McCabe) (cmccabe: rev fc2ed4a1f9a19d61f5e3cb4fd843604f0c7fe95f) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          rkanter Robert Kanter added a comment -

          Looks like this broke a TestRollingLevelDBTimelineStore: YARN-3812

          Show
          rkanter Robert Kanter added a comment - Looks like this broke a TestRollingLevelDBTimelineStore : YARN-3812
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for pointing that out, Robert Kanter. The problem seems to be in ImmutableFsPermission... I commented on the JIRA (as did Jason Lowe).

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for pointing that out, Robert Kanter . The problem seems to be in ImmutableFsPermission ... I commented on the JIRA (as did Jason Lowe ).

            People

            • Assignee:
              varun_saxena Varun Saxena
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development