Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9491

Tests should get the number of pending async delets via FsDatasetTestUtils

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:

      Description

      A few unit tests use DataNodeTestUtils#getPendingAsyncDeletions to retrieve the number of pending async deletions. It internally calls FsDatasetTestUtil#getPendingAsyncDeletions:

      public static long getPendingAsyncDeletions(FsDatasetSpi<?> fsd) {
          return ((FsDatasetImpl)fsd).asyncDiskService.countPendingDeletions();
      }
      

      This assumes FsDatasetImpl is (the only implementation of) FsDataset. However FsDataset is pluggable and can have other implementations.

      We can abstract getting the number of async deletions in FsDatasetTestUtils.

      1. HDFS-9491.001.patch
        5 kB
        Tony Wu
      2. HDFS-9491.002.patch
        5 kB
        Tony Wu

        Activity

        Hide
        twu Tony Wu added a comment -

        In v1 patch:

        • Add FsDatasetTestUtils#getPendingAsyncDeletions.
        • Update relevant tests to use the new API.
        • Removed old APIs.
        Show
        twu Tony Wu added a comment - In v1 patch: Add FsDatasetTestUtils#getPendingAsyncDeletions . Update relevant tests to use the new API. Removed old APIs.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        +1 mvninstall 7m 46s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_66
        +1 compile 0m 40s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_66
        +1 javac 0m 38s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_85
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_85
        -1 unit 52m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 49m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        128m 59s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestWriteReadStripedFile
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775106/HDFS-9491.001.patch
        JIRA Issue HDFS-9491
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3e4ee3c99384 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1cc7e61
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13717/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13717/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 46s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 47s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 +1 javac 0m 38s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_85 +1 javac 0m 41s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_85 -1 unit 52m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 49m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 128m 59s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775106/HDFS-9491.001.patch JIRA Issue HDFS-9491 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3e4ee3c99384 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1cc7e61 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13717/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13717/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13717/console This message was automatically generated.
        Hide
        twu Tony Wu added a comment -

        Manually ran the failed tests: TestWriteReadStripedFile, TestRenameWithSnapshots & TestDFSStripedOutputStreamWithFailure130 with JDK 1.8 and 1.7 on OSX. All tests pass without error. None of the failed tests use getPendingAsyncDeletions so it should not be related to the patch.

        ASF License warning is also not related to the patch as it does not include any license related change.

        Show
        twu Tony Wu added a comment - Manually ran the failed tests: TestWriteReadStripedFile, TestRenameWithSnapshots & TestDFSStripedOutputStreamWithFailure130 with JDK 1.8 and 1.7 on OSX. All tests pass without error. None of the failed tests use getPendingAsyncDeletions so it should not be related to the patch. ASF License warning is also not related to the patch as it does not include any license related change.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Hi, Tony Wu

        This patch can not compile on trunk

        [ERROR] location: interface org.apache.hadoop.hdfs.server.datanode.FsDatasetTestUtils
        [ERROR] /Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java:[381,3] method does not override or implement a method from a supertype
        [ERROR] /Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java:[104,48] cannot find symbol
        

        Could you fix it and re-submit your patch? Thanks

        Show
        eddyxu Lei (Eddy) Xu added a comment - Hi, Tony Wu This patch can not compile on trunk [ERROR] location: interface org.apache.hadoop.hdfs.server.datanode.FsDatasetTestUtils [ERROR] /Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java:[381,3] method does not override or implement a method from a supertype [ERROR] /Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java:[104,48] cannot find symbol Could you fix it and re-submit your patch? Thanks
        Hide
        twu Tony Wu added a comment -

        In v2 patch:

        • Rebased to latest trunk.
        • Verified the build completes on OSX & Linux.
        • Verified the following tests (they all use the modified API) pass on OSX & Linux: TestDecommission, TestRBWBlockInvalidation, TestDNFencing, TestHASafeMode, TestLazyWriter.
        Show
        twu Tony Wu added a comment - In v2 patch: Rebased to latest trunk. Verified the build completes on OSX & Linux. Verified the following tests (they all use the modified API) pass on OSX & Linux: TestDecommission, TestRBWBlockInvalidation, TestDNFencing, TestHASafeMode, TestLazyWriter.
        Hide
        twu Tony Wu added a comment -

        Hi Lei (Eddy) Xu,

        I Just rebased the patch on latest trunk and ran a few tests. It was able to build correctly and the selected tests also ran without issue. Please take a look at the v2 patch.

        Thanks,
        Tony

        Show
        twu Tony Wu added a comment - Hi Lei (Eddy) Xu , I Just rebased the patch on latest trunk and ran a few tests. It was able to build correctly and the selected tests also ran without issue. Please take a look at the v2 patch. Thanks, Tony
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        +1 mvninstall 8m 33s trunk passed
        +1 compile 0m 47s trunk passed with JDK v1.8.0_66
        +1 compile 0m 45s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 5s trunk passed
        +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 54s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.8.0_66
        +1 javac 0m 46s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.7.0_85
        +1 javac 0m 44s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 15s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 56s the patch passed with JDK v1.7.0_85
        -1 unit 56m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 21s Patch generated 58 ASF License warnings.
        139m 54s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestReplication
          hadoop.hdfs.TestEncryptionZonesWithKMS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775831/HDFS-9491.002.patch
        JIRA Issue HDFS-9491
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a3359553efc7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cbc7b6b
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13772/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13772/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 33s trunk passed +1 compile 0m 47s trunk passed with JDK v1.8.0_66 +1 compile 0m 45s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 54s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 52s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_66 +1 javac 0m 46s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_85 +1 javac 0m 44s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 56s the patch passed with JDK v1.7.0_85 -1 unit 56m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 21s Patch generated 58 ASF License warnings. 139m 54s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestReplication   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775831/HDFS-9491.002.patch JIRA Issue HDFS-9491 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a3359553efc7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cbc7b6b findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13772/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13772/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13772/console This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Hi, Tony

        TestDecommission failed on JDK 8, could you help to verify that?

        Show
        eddyxu Lei (Eddy) Xu added a comment - Hi, Tony TestDecommission failed on JDK 8, could you help to verify that?
        Hide
        twu Tony Wu added a comment -

        Hi Lei (Eddy) Xu,

        I just ran TestDecommission with the patch with JDK 8 on OSX, the test passes without error.

        The failed test TestDecommission is not related to the patch. Please see the error log from Jenkins is below:

        Tests run: 19, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 160.801 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestDecommission
        testDecommissionWithOpenfile(org.apache.hadoop.hdfs.TestDecommission)  Time elapsed: 1.753 sec  <<< ERROR!
        java.lang.RuntimeException: Error while running command to get file permissions : ExitCodeException exitCode=127: /bin/ls: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Permission denied
        
        	at org.apache.hadoop.util.Shell.runCommand(Shell.java:927)
        	at org.apache.hadoop.util.Shell.run(Shell.java:838)
        	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1117)
        	at org.apache.hadoop.util.Shell.execCommand(Shell.java:1211)
        	at org.apache.hadoop.util.Shell.execCommand(Shell.java:1193)
        	at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1081)
        	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:702)
        	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:677)
        	at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:155)
        	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:172)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2459)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2501)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2484)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2376)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1592)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:844)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:482)
        	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
        	at org.apache.hadoop.hdfs.TestDecommission.startCluster(TestDecommission.java:334)
        	at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithOpenfile(TestDecommission.java:830)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:497)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
        
        	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:742)
        	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:677)
        	at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:155)
        	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:172)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2459)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2501)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2484)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2376)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1592)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:844)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:482)
        	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
        	at org.apache.hadoop.hdfs.TestDecommission.startCluster(TestDecommission.java:334)
        	at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithOpenfile(TestDecommission.java:830)
        

        The test case in TestDecommission that uses the changed API is testDecommissionOnStandby, which passes in the Hadoop QA Jenkins run. There seems to be some system issue with the Jenkins servers.

        Thanks,
        Tony

        Show
        twu Tony Wu added a comment - Hi Lei (Eddy) Xu , I just ran TestDecommission with the patch with JDK 8 on OSX, the test passes without error. The failed test TestDecommission is not related to the patch. Please see the error log from Jenkins is below: Tests run: 19, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 160.801 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestDecommission testDecommissionWithOpenfile(org.apache.hadoop.hdfs.TestDecommission) Time elapsed: 1.753 sec <<< ERROR! java.lang.RuntimeException: Error while running command to get file permissions : ExitCodeException exitCode=127: /bin/ls: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Permission denied at org.apache.hadoop.util.Shell.runCommand(Shell.java:927) at org.apache.hadoop.util.Shell.run(Shell.java:838) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1117) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1211) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1193) at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1081) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:702) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:677) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:155) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:172) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2459) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2501) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2484) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2376) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1592) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:844) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:482) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441) at org.apache.hadoop.hdfs.TestDecommission.startCluster(TestDecommission.java:334) at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithOpenfile(TestDecommission.java:830) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:742) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:677) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:155) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:172) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2459) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2501) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2484) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2376) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1592) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:844) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:482) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441) at org.apache.hadoop.hdfs.TestDecommission.startCluster(TestDecommission.java:334) at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithOpenfile(TestDecommission.java:830) The test case in TestDecommission that uses the changed API is testDecommissionOnStandby , which passes in the Hadoop QA Jenkins run. There seems to be some system issue with the Jenkins servers. Thanks, Tony
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        +1. I ran related tests locally without errors.

        Thanks very much, Tony Wu.

        Show
        eddyxu Lei (Eddy) Xu added a comment - +1. I ran related tests locally without errors. Thanks very much, Tony Wu .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8927 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8927/)
        HDFS-9491. Tests should get the number of pending async deletes via (lei: rev 4265a85f6d9fbf48cdd741bd3d27bfced52f34ca)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8927 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8927/ ) HDFS-9491 . Tests should get the number of pending async deletes via (lei: rev 4265a85f6d9fbf48cdd741bd3d27bfced52f34ca) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #668 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/668/)
        HDFS-9491. Tests should get the number of pending async deletes via (lei: rev 4265a85f6d9fbf48cdd741bd3d27bfced52f34ca)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #668 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/668/ ) HDFS-9491 . Tests should get the number of pending async deletes via (lei: rev 4265a85f6d9fbf48cdd741bd3d27bfced52f34ca) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            twu Tony Wu
            Reporter:
            twu Tony Wu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development