Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9042

Add a test for umask in FileSystemContractBaseTest

    Details

    • Type: Test
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add a unit test to make sure umask is working correctly in FileSystemContractBaseTest.

      1. HADOOP-9042.008.patch
        2 kB
        Colin P. McCabe
      2. HADOOP-9042.009.patch
        4 kB
        Colin P. McCabe
      3. HADOOP-9042.010.patch
        3 kB
        Colin P. McCabe
      4. HDFS-3975.001.patch
        2 kB
        Colin P. McCabe
      5. HDFS-3975.002.patch
        2 kB
        Colin P. McCabe
      6. HDFS-3975.003.patch
        2 kB
        Colin P. McCabe
      7. HDFS-3975.004.patch
        2 kB
        Colin P. McCabe
      8. HDFS-3975.005.patch
        2 kB
        Colin P. McCabe
      9. HDFS-3975.006.patch
        2 kB
        Colin P. McCabe

        Issue Links

          Activity

          Hide
          tlipcon Todd Lipcon added a comment -

          +1 pending jenkins

          Show
          tlipcon Todd Lipcon added a comment - +1 pending jenkins
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546595/HDFS-3975.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3233//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3233//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546595/HDFS-3975.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3233//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3233//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Are you sure that the test is not already covered by TestWebHdfsFileSystemContract and TestFSMainOperationsWebHdfs?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Are you sure that the test is not already covered by TestWebHdfsFileSystemContract and TestFSMainOperationsWebHdfs?
          Hide
          cmccabe Colin P. McCabe added a comment -

          Are you sure that the test is not already covered by TestWebHdfsFileSystemContract and TestFSMainOperationsWebHdfs?

          That's a good question. I can see that there's a few places where we test permissions-- for example in TestWebHdfsFileSystemContract#testResponseCode we create a directory and verify that it got permissions of 0755. This implicitly creates the directory with 0777, and since the default umask is 022, it gets permissions 0755.

          I'm not sure if that really qualifies as a test of umask though-- we never explicitly set umask to something other than the default. In fact we never explicitly specify a non-default permission mode when creating a directory or file in the existing webhdfs tests, so I would say that existing coverage not very complete.

          What do you think? Would it make more sense to add this new test in TestWebHdfsFileSystemContract?

          Show
          cmccabe Colin P. McCabe added a comment - Are you sure that the test is not already covered by TestWebHdfsFileSystemContract and TestFSMainOperationsWebHdfs? That's a good question. I can see that there's a few places where we test permissions-- for example in TestWebHdfsFileSystemContract#testResponseCode we create a directory and verify that it got permissions of 0755 . This implicitly creates the directory with 0777 , and since the default umask is 022 , it gets permissions 0755 . I'm not sure if that really qualifies as a test of umask though-- we never explicitly set umask to something other than the default. In fact we never explicitly specify a non-default permission mode when creating a directory or file in the existing webhdfs tests, so I would say that existing coverage not very complete. What do you think? Would it make more sense to add this new test in TestWebHdfsFileSystemContract?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          It does make more sense to add the umask test in FileSystemContractBaseTest. Then, it will also test DistributedFileSystem.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - It does make more sense to add the umask test in FileSystemContractBaseTest. Then, it will also test DistributedFileSystem.
          Hide
          cmccabe Colin P. McCabe added a comment -
          • move umask test to FileSystemContractBase, since it's more general than just webhdfs
          Show
          cmccabe Colin P. McCabe added a comment - move umask test to FileSystemContractBase , since it's more general than just webhdfs
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546890/HDFS-3975.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3243//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3243//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546890/HDFS-3975.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3243//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3243//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          skip umask checks for S3, S3n, since these filesystems do not yet implement permissions

          Show
          cmccabe Colin P. McCabe added a comment - skip umask checks for S3, S3n, since these filesystems do not yet implement permissions
          Hide
          cmccabe Colin P. McCabe added a comment -

          can't test umask for s3, s3n yet since it is not implemented!

          Show
          cmccabe Colin P. McCabe added a comment - can't test umask for s3, s3n yet since it is not implemented!
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546895/HDFS-3975.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3244//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3244//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546895/HDFS-3975.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3244//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3244//console This message is automatically generated.
          Hide
          eli Eli Collins added a comment -

          +1, looks good. Btw you can implement the initial test using:

          Assume.assumeTrue(!fs.getScheme().equals("s3") && !fs.getScheme().equals("s3n"))
          
          Show
          eli Eli Collins added a comment - +1, looks good. Btw you can implement the initial test using: Assume.assumeTrue(!fs.getScheme().equals( "s3" ) && !fs.getScheme().equals( "s3n" ))
          Hide
          cmccabe Colin P. McCabe added a comment -

          use "assume" as Eli suggested

          Show
          cmccabe Colin P. McCabe added a comment - use "assume" as Eli suggested
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553218/HDFS-3975.004.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3490//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3490//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553218/HDFS-3975.004.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3490//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3490//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          it looks like Assume doesn't work with junit3. So use a regular old if statement.

          Show
          cmccabe Colin P. McCabe added a comment - it looks like Assume doesn't work with junit3. So use a regular old if statement.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553422/HDFS-3975.005.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3497//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3497//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553422/HDFS-3975.005.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3497//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3497//console This message is automatically generated.
          Hide
          eli Eli Collins added a comment -

          Can remove the import of Assume or convert the test to junit4, otherwise +1

          Show
          eli Eli Collins added a comment - Can remove the import of Assume or convert the test to junit4, otherwise +1
          Hide
          cmccabe Colin P. McCabe added a comment -

          remove 'import assume'

          Show
          cmccabe Colin P. McCabe added a comment - remove 'import assume'
          Hide
          eli Eli Collins added a comment -

          +1 I've committed this and merged to branch-2.

          Show
          eli Eli Collins added a comment - +1 I've committed this and merged to branch-2.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3021 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3021/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3021 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3021/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          kihwal Kihwal Lee added a comment -

          This HDFS test case is failing. The precommit build did not run any hdfs test, so it didn't catch this.

          Running org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          Tests run: 30, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.774 sec <<< FAILURE!
          testMkdirsWithUmask(org.apache.hadoop.hdfs.TestHDFSFileSystemContract) Time elapsed: 1046 sec <<< FAILURE!
          junit.framework.AssertionFailedError: expected:<461> but was:<493>
          at junit.framework.Assert.fail(Assert.java:47)
          at junit.framework.Assert.failNotEquals(Assert.java:283)
          at junit.framework.Assert.assertEquals(Assert.java:64)
          at junit.framework.Assert.assertEquals(Assert.java:182)
          at junit.framework.Assert.assertEquals(Assert.java:188)
          at org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:170)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at junit.framework.TestCase.runTest(TestCase.java:168)
          at junit.framework.TestCase.runBare(TestCase.java:134)
          at junit.framework.TestResult$1.protect(TestResult.java:110)
          at junit.framework.TestResult.runProtected(TestResult.java:128)
          at junit.framework.TestResult.run(TestResult.java:113)
          at junit.framework.TestCase.run(TestCase.java:124)
          at junit.framework.TestSuite.runTest(TestSuite.java:243)
          at junit.framework.TestSuite.run(TestSuite.java:238)
          at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
          at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
          at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
          at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
          at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
          at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
          at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
          at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

          Show
          kihwal Kihwal Lee added a comment - This HDFS test case is failing. The precommit build did not run any hdfs test, so it didn't catch this. Running org.apache.hadoop.hdfs.TestHDFSFileSystemContract Tests run: 30, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.774 sec <<< FAILURE! testMkdirsWithUmask(org.apache.hadoop.hdfs.TestHDFSFileSystemContract) Time elapsed: 1046 sec <<< FAILURE! junit.framework.AssertionFailedError: expected:<461> but was:<493> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:182) at junit.framework.Assert.assertEquals(Assert.java:188) at org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
          Hide
          cmccabe Colin P. McCabe added a comment -

          The problem seems to be that DFSClient caches the value of fs.permissions.umask-mode once when it's created, and doesn't update it if the configured value changes. The value is cached in a final field in DFSClient#Conf#uMask.

          Show
          cmccabe Colin P. McCabe added a comment - The problem seems to be that DFSClient caches the value of fs.permissions.umask-mode once when it's created, and doesn't update it if the configured value changes. The value is cached in a final field in DFSClient#Conf#uMask .
          Hide
          cmccabe Colin P. McCabe added a comment -

          here's a fix.

          Show
          cmccabe Colin P. McCabe added a comment - here's a fix.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #37 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/37/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #37 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/37/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1227 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1227/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1227 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1227/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1258 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1258/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1258 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1258/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Colin Patrick McCabe (Revision 1409635) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409635 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          eli Eli Collins added a comment -

          I reverted the change, Colin, please post a new patch that incorporates your fix.

          Btw looking at the delta why define TEST_UMASK in common if it's only used in HDFS?

          Show
          eli Eli Collins added a comment - I reverted the change, Colin, please post a new patch that incorporates your fix. Btw looking at the delta why define TEST_UMASK in common if it's only used in HDFS?
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3025/)
          Revert HADOOP-9042. (Revision 1409902)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3025/ ) Revert HADOOP-9042 . (Revision 1409902) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          cmccabe Colin P. McCabe added a comment -

          here's an updated patch which includes the fix.

          To answer your question, Eli, TEST_UMASK is used in both FileSystemContractBaseTest and TestHDFSFileSystemContract. The issue is that HDFS requires the default umask to be set when the FileSystem is created, whereas the other filesystems can accept it being changed later.

          Show
          cmccabe Colin P. McCabe added a comment - here's an updated patch which includes the fix. To answer your question, Eli, TEST_UMASK is used in both FileSystemContractBaseTest and TestHDFSFileSystemContract . The issue is that HDFS requires the default umask to be set when the FileSystem is created, whereas the other filesystems can accept it being changed later.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553678/HADOOP-9042.009.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1757//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553678/HADOOP-9042.009.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1757//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/)
          Revert HADOOP-9042. (Revision 1409902)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/ ) Revert HADOOP-9042 . (Revision 1409902) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/)
          Revert HADOOP-9042. (Revision 1409902)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/ ) Revert HADOOP-9042 . (Revision 1409902) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/)
          Revert HADOOP-9042. (Revision 1409902)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/ ) Revert HADOOP-9042 . (Revision 1409902) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409902 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553840/HADOOP-9042.009.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1765//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553840/HADOOP-9042.009.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1765//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553855/HADOOP-9042.010.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1766//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1766//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553855/HADOOP-9042.010.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1766//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1766//console This message is automatically generated.
          Hide
          eli Eli Collins added a comment -

          +1 latest patch looks good

          Show
          eli Eli Collins added a comment - +1 latest patch looks good
          Hide
          eli Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Colin.

          Show
          eli Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3049 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3049/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3049 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3049/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #43 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/43/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #43 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/43/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1233/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1233/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1264 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1264/)
          HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1264 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1264/ ) HADOOP-9042 . Add a test for umask in FileSystemContractBaseTest. Contributed by Colin McCabe (Revision 1411879) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411879 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSFileSystemContract.java

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development