Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13787

Azure testGlobStatusThrowsExceptionForUnreadableDir fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: fs/azure
    • Labels:
      None

      Description

      Test TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir failed in trunk:

      Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<< FAILURE! - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
      testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked) Time elapsed: 1.111 sec <<< FAILURE!
      java.lang.AssertionError: Should throw IOException
      at org.junit.Assert.fail(Assert.java:88)
      at org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
      Results :
      Failed tests:
      TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643 Should throw IOException
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
      

        Issue Links

          Activity

          Hide
          jzhuge John Zhuge added a comment -

          NativeAzureFileSystem does not support permission checking even though file permission FileMetadata#permissionStatus is stored in the metadata store AzureNativeFileSystemStore.

          I will disable this test for NativeAzureFileSystem.

          Show
          jzhuge John Zhuge added a comment - NativeAzureFileSystem does not support permission checking even though file permission FileMetadata#permissionStatus is stored in the metadata store AzureNativeFileSystemStore . I will disable this test for NativeAzureFileSystem.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001:

          • Disable testGlobStatusThrowsExceptionForUnreadableDir in TestNativeAzureFileSystemOperationsMocked, just as testListStatusThrowsExceptionForUnreadableDir.
          Show
          jzhuge John Zhuge added a comment - Patch 001: Disable testGlobStatusThrowsExceptionForUnreadableDir in TestNativeAzureFileSystemOperationsMocked , just as testListStatusThrowsExceptionForUnreadableDir .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 26s trunk passed
          +1 compile 0m 22s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 37s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 1m 36s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          16m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13787
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836724/HADOOP-13787.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a34db7a4b94a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7e521c5
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 26s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 1m 36s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 16m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13787 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836724/HADOOP-13787.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a34db7a4b94a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7e521c5 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge for the quick fix! +1, committing shortly.

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge for the quick fix! +1, committing shortly.
          Hide
          xiaochen Xiao Chen added a comment -

          Committed to trunk. Thanks John Zhuge for the fix and Mingliang Liu for reporting the issue.

          Hi John, since the impact of that change is big, could you run a full hadoop test at that commit to double check no other tests are broken before v.s. after? Thanks a lot.

          Show
          xiaochen Xiao Chen added a comment - Committed to trunk. Thanks John Zhuge for the fix and Mingliang Liu for reporting the issue. Hi John, since the impact of that change is big, could you run a full hadoop test at that commit to double check no other tests are broken before v.s. after? Thanks a lot.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks @xiao chen for the review and commit! And Mingliang Liu for reporting.

          Sure, I will run a full Hadoop test to compare before and after HADOOP-7352. Since it will take a long time, I will run the full test on my dist_test cluster that supports Java 8 and Hadoop trunk.

          Show
          jzhuge John Zhuge added a comment - Thanks @xiao chen for the review and commit! And Mingliang Liu for reporting. Sure, I will run a full Hadoop test to compare before and after HADOOP-7352 . Since it will take a long time, I will run the full test on my dist_test cluster that supports Java 8 and Hadoop trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10765 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10765/)
          HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. (xiao: rev 20c4d8efab425fe3893ef9a5bec85d4a3b64d5a6)

          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemOperationsMocked.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10765 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10765/ ) HADOOP-13787 . Azure testGlobStatusThrowsExceptionForUnreadableDir fails. (xiao: rev 20c4d8efab425fe3893ef9a5bec85d4a3b64d5a6) (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemOperationsMocked.java
          Hide
          xiaochen Xiao Chen added a comment -

          SGTM, thanks John Zhuge.

          Show
          xiaochen Xiao Chen added a comment - SGTM, thanks John Zhuge .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          this'll need to be be done for Swift too —sorry. Welcome to the world of object store-related regressions

          Show
          stevel@apache.org Steve Loughran added a comment - this'll need to be be done for Swift too —sorry. Welcome to the world of object store-related regressions
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Steve Loughran for the catch. Filed HADOOP-13795. With that, we should have covered all derived classes of FSMainOperationsBaseTest. Let me know anything is missing.

          Show
          jzhuge John Zhuge added a comment - Thanks Steve Loughran for the catch. Filed HADOOP-13795 . With that, we should have covered all derived classes of FSMainOperationsBaseTest . Let me know anything is missing.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Steve for reporting! Hopefully John's full unit test run will find out whatever is left there....

          Show
          xiaochen Xiao Chen added a comment - Thanks Steve for reporting! Hopefully John's full unit test run will find out whatever is left there....
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ...think so; an S3a implementation of that is still an open issue. Linking to HADOOP-13373 so that whoever implements that, knows to make sure that this override is done.

          Show
          stevel@apache.org Steve Loughran added a comment - ...think so; an S3a implementation of that is still an open issue. Linking to HADOOP-13373 so that whoever implements that, knows to make sure that this override is done.
          Hide
          jzhuge John Zhuge added a comment -

          Here is the comparison test result:

          • The current trunk is 0aafc12, compared to 29caf6d, the commit right before HADOOP-7352.
          • All 173 hadoop-tools tests passed.
          • 5 out of 441 hadoop-common tests failed, 3 regressions.
          • I will file JIRAs for both regressions and non-regressions.

          org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem

          • TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288 Should throw IOException
          • (regression) TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643 Should throw IOException

          org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem

          • TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288 Should throw IOException
          • (regression) TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643 Should throw IOException

          TestLocalDirAllocator

          • TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109 Checking for build/test/temp/RELATIVE2 in build/test/temp/RELATIVE1/block6738757787047387788.tmp - FAILED!
          • TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking for build/test/temp/RELATIVE1 in build/test/temp/RELATIVE0/block125615631432807097.tmp - FAILED!
          • TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109 Checking for /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block3679320121221680948.tmp - FAILED!
          • TestLocalDirAllocator.test0:141->validateTempDirCreation:109 Checking for /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block5094057430925940349.tmp - FAILED!
          • TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109 Checking for file:/tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block959204179043794136.tmp - FAILED!
          • TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking for file:/tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block8256098597810969453.tmp - FAILED!

          TestPathData

          • (regression) TestPathData.testGlobThrowsExceptionForUnreadableDir:230 Should throw IOException

          TestRollingFileSystemSinkWithLocal

          • TestRollingFileSystemSinkWithLocal.testFailedWrite:116 No exception was generated while writing metrics even though the target directory was not writable
          Show
          jzhuge John Zhuge added a comment - Here is the comparison test result: The current trunk is 0aafc12 , compared to 29caf6d , the commit right before HADOOP-7352 . All 173 hadoop-tools tests passed. 5 out of 441 hadoop-common tests failed, 3 regressions. I will file JIRAs for both regressions and non-regressions. org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288 Should throw IOException (regression) TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643 Should throw IOException org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288 Should throw IOException (regression) TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643 Should throw IOException TestLocalDirAllocator TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109 Checking for build/test/temp/RELATIVE2 in build/test/temp/RELATIVE1/block6738757787047387788.tmp - FAILED! TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking for build/test/temp/RELATIVE1 in build/test/temp/RELATIVE0/block125615631432807097.tmp - FAILED! TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109 Checking for /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block3679320121221680948.tmp - FAILED! TestLocalDirAllocator.test0:141->validateTempDirCreation:109 Checking for /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block5094057430925940349.tmp - FAILED! TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109 Checking for file:/tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block959204179043794136.tmp - FAILED! TestLocalDirAllocator.test0:140->validateTempDirCreation:109 Checking for file:/tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1 in /tmp/run_tha_testQ8gxo9/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block8256098597810969453.tmp - FAILED! TestPathData (regression) TestPathData.testGlobThrowsExceptionForUnreadableDir:230 Should throw IOException TestRollingFileSystemSinkWithLocal TestRollingFileSystemSinkWithLocal.testFailedWrite:116 No exception was generated while writing metrics even though the target directory was not writable
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John! Will follow up on all related jiras.
          Not sure why the hadoop-common ones were missed in pre-commit - Jenkins gave +1 in HADOOP-7352.

          Show
          xiaochen Xiao Chen added a comment - Thanks John! Will follow up on all related jiras. Not sure why the hadoop-common ones were missed in pre-commit - Jenkins gave +1 in HADOOP-7352 .
          Hide
          jzhuge John Zhuge added a comment - - edited

          Forgot to mention I ran the tests on grind. It turned out 4 of 5 tests are flaky. They did succeed after a few retries using grind. Only TestLocalDirAllocator failed on grind consistently.

          All 5 tests passed mvn test on Ubuntu 14. This explains the passed pre-commit tests for HADOOP-7352.

          Show
          jzhuge John Zhuge added a comment - - edited Forgot to mention I ran the tests on grind. It turned out 4 of 5 tests are flaky. They did succeed after a few retries using grind. Only TestLocalDirAllocator failed on grind consistently. All 5 tests passed mvn test on Ubuntu 14. This explains the passed pre-commit tests for HADOOP-7352 .

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development