Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9493

Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:

      Description

      Tested in both Gentoo Linux and Mac.

      -------------------------------------------------------
      T E S T S
      -------------------------------------------------------
      Running org.apache.hadoop.hdfs.server.namenode.TestMetaSave
      Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 34.159 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestMetaSave
      testMetasaveAfterDelete(org.apache.hadoop.hdfs.server.namenode.TestMetaSave) Time elapsed: 15.318 sec <<< FAILURE!
      java.lang.AssertionError: null
      at org.junit.Assert.fail(Assert.java:86)
      at org.junit.Assert.assertTrue(Assert.java:41)
      at org.junit.Assert.assertTrue(Assert.java:52)
      at org.apache.hadoop.hdfs.server.namenode.TestMetaSave.testMetasaveAfterDelete(TestMetaSave.java:154)

      1. HDFS-9493.001.patch
        4 kB
        Tony Wu
      2. HDFS-9493.002.patch
        6 kB
        Tony Wu
      3. HDFS-9493.003.patch
        6 kB
        Tony Wu

        Issue Links

          Activity

          Hide
          twu Tony Wu added a comment -

          Hi Mingliang Liu, I would like to work on fixing this test.

          Did some analysis on the failure by printing out the metasave content. Turns out the metasave output for the current test contains 2 Datanodes:

          metasave out: 1 files and directories, 0 blocks = 1 total filesystem objects
          metasave out: Live Datanodes: 1
          metasave out: Dead Datanodes: 1
          metasave out: Metasave: Blocks waiting for replication: 0
          metasave out: Mis-replicated blocks that have been postponed:
          metasave out: Metasave: Blocks being replicated: 0
          metasave out: Metasave: Blocks 4 waiting deletion from 2 datanodes.
          metasave out: 127.0.0.1:53465
          metasave out: LightWeightHashSet(size=2, modification=2, entries.length=16)
          metasave out: 127.0.0.1:53469
          metasave out: LightWeightHashSet(size=2, modification=2, entries.length=16)
          metasave out: Metasave: Number of datanodes: 2
          metasave out: 127.0.0.1:53465 IN 998093619200(929.55 GB) 10270(10.03 KB) 0.00% 882663514112(822.04 GB) 0(0 B) 0(0 B) 100.00% 0(0 B) Fri Dec 11 17:48:41 PST 2015
          metasave out: 127.0.0.1:53469 IN 998093619200(929.55 GB) 8192(8 KB) 0.00% 882663825408(822.04 GB) 0(0 B) 0(0 B) 100.00% 0(0 B) Fri Dec 11 17:48:26 PST 2015
          

          This leads me to believe the following wait time was not long enough:

              // wait for namenode to discover that a datanode is dead
              Thread.sleep(15000);
          

          After increasing the sleep time to 30 seconds, the test was able to pass consistently.

          The invalid bock count shown in Block x waiting deletion... statement is updated by blockManager.removeBlocksAssociatedTo(), which is called by DatanodeManager#removeDeadDatanode(). This only happens at HeartbeatManager#heartbeatCheck(). Using sleep may not be the best way to ensure the Datanode is deleted by Namenode.

          I will upload a patch with a more robust way of waiting for the Datanode to be removed, instead of relying on Thread.sleep().

          Show
          twu Tony Wu added a comment - Hi Mingliang Liu , I would like to work on fixing this test. Did some analysis on the failure by printing out the metasave content. Turns out the metasave output for the current test contains 2 Datanodes: metasave out: 1 files and directories, 0 blocks = 1 total filesystem objects metasave out: Live Datanodes: 1 metasave out: Dead Datanodes: 1 metasave out: Metasave: Blocks waiting for replication: 0 metasave out: Mis-replicated blocks that have been postponed: metasave out: Metasave: Blocks being replicated: 0 metasave out: Metasave: Blocks 4 waiting deletion from 2 datanodes. metasave out: 127.0.0.1:53465 metasave out: LightWeightHashSet(size=2, modification=2, entries.length=16) metasave out: 127.0.0.1:53469 metasave out: LightWeightHashSet(size=2, modification=2, entries.length=16) metasave out: Metasave: Number of datanodes: 2 metasave out: 127.0.0.1:53465 IN 998093619200(929.55 GB) 10270(10.03 KB) 0.00% 882663514112(822.04 GB) 0(0 B) 0(0 B) 100.00% 0(0 B) Fri Dec 11 17:48:41 PST 2015 metasave out: 127.0.0.1:53469 IN 998093619200(929.55 GB) 8192(8 KB) 0.00% 882663825408(822.04 GB) 0(0 B) 0(0 B) 100.00% 0(0 B) Fri Dec 11 17:48:26 PST 2015 This leads me to believe the following wait time was not long enough: // wait for namenode to discover that a datanode is dead Thread .sleep(15000); After increasing the sleep time to 30 seconds, the test was able to pass consistently. The invalid bock count shown in Block x waiting deletion... statement is updated by blockManager.removeBlocksAssociatedTo() , which is called by DatanodeManager#removeDeadDatanode() . This only happens at HeartbeatManager#heartbeatCheck() . Using sleep may not be the best way to ensure the Datanode is deleted by Namenode. I will upload a patch with a more robust way of waiting for the Datanode to be removed, instead of relying on Thread.sleep() .
          Hide
          twu Tony Wu added a comment -

          In v1 patch:

          • Add a helper function BlockManagerTestUtil.isDatanodeRemoved() to check if DN is removed by NN.
          • Update TestMetaSave#testMetasaveAfterDelete() to use the helper function.

          Verified on OSX and Linux (CentOS 6) the test passes consistently.

          Show
          twu Tony Wu added a comment - In v1 patch: Add a helper function BlockManagerTestUtil.isDatanodeRemoved() to check if DN is removed by NN. Update TestMetaSave#testMetasaveAfterDelete() to use the helper function. Verified on OSX and Linux (CentOS 6) the test passes consistently.
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Tony Wu,

          Thanks for working on this. I think you analysis is correct that the datanode (DN) is not marked as dead when saving metadata.

          I like your idea to make the code robust by waiting for the DN to be removed. Sleeping for 15 seconds is not able to guarantee the DN is already removed (problem 1). Plus, 15 seconds idle time is too long for a unit test (problem 2). There is another bug in the code that stops DN in both testMetaSave() and testMetasaveAfterDelete(). Both of the test assume there are two live DNs in cluster and remove the second one before testing metaSave (problem 3). The latter should fail if the DN is removed by the former test.

          Your patch should have solved the first problem, but still, it needs wait 10~20 seconds on my local machine before the DN is removed. To solve the second problem, we can expire a DN heartbeat on the NN in MiniDFSCluster via setDataNodeDead(). As to the third problem, we can either build a cluster for each test, or add the DN back after testing metaSave.

          Added Tao Luo and Konstantin Shvachko to the watchers list as they worked on testMetasaveAfterDelete() in HDFS-4878.

          Show
          liuml07 Mingliang Liu added a comment - Hi Tony Wu , Thanks for working on this. I think you analysis is correct that the datanode (DN) is not marked as dead when saving metadata. I like your idea to make the code robust by waiting for the DN to be removed. Sleeping for 15 seconds is not able to guarantee the DN is already removed (problem 1). Plus, 15 seconds idle time is too long for a unit test (problem 2). There is another bug in the code that stops DN in both testMetaSave() and testMetasaveAfterDelete() . Both of the test assume there are two live DNs in cluster and remove the second one before testing metaSave (problem 3). The latter should fail if the DN is removed by the former test. Your patch should have solved the first problem, but still, it needs wait 10~20 seconds on my local machine before the DN is removed. To solve the second problem, we can expire a DN heartbeat on the NN in MiniDFSCluster via setDataNodeDead() . As to the third problem, we can either build a cluster for each test, or add the DN back after testing metaSave . Added Tao Luo and Konstantin Shvachko to the watchers list as they worked on testMetasaveAfterDelete() in HDFS-4878 .
          Hide
          twu Tony Wu added a comment -

          Hi Mingliang Liu,

          Thanks for your detailed comments. I have confirmed problem 2 & 3 after revisiting the test code.

          For problem 3, I mis-read the @BeforeClass and @AfterClass tags, thinking MiniDFSCluster will be torn down and rebuilt for every test. That is not the case here. Instead as you said by the time testMetaSave() runs testMetaSaveAfterDelete() would have already removed the DN and the cluster.stopDataNode() call in the second test is essentially a no-op. Looks like the tests just happened to be working. The setup and tearDown functions should have been executed before and after each test case.

          For problem 1 and 2, to further reduce the wait time for this unit test, BlockManagerTestUtil#noticeDeadDatanode() will be useful. This helper function will set the DN to be dead right away instead of waiting for HB timeout.

          I will rework the current patch and incorporate the findings above.

          Show
          twu Tony Wu added a comment - Hi Mingliang Liu , Thanks for your detailed comments. I have confirmed problem 2 & 3 after revisiting the test code. For problem 3, I mis-read the @BeforeClass and @AfterClass tags, thinking MiniDFSCluster will be torn down and rebuilt for every test. That is not the case here. Instead as you said by the time testMetaSave() runs testMetaSaveAfterDelete() would have already removed the DN and the cluster.stopDataNode() call in the second test is essentially a no-op. Looks like the tests just happened to be working. The setup and tearDown functions should have been executed before and after each test case. For problem 1 and 2, to further reduce the wait time for this unit test, BlockManagerTestUtil#noticeDeadDatanode() will be useful. This helper function will set the DN to be dead right away instead of waiting for HB timeout. I will rework the current patch and incorporate the findings above.
          Hide
          twu Tony Wu added a comment -

          In v2 patch:

          • Use BlockManagerTestUtil#noticeDeadDatanode() to reduce the test run time. After this call the NN will declare the DN dead right away, rather than waiting for HB timeout.
          • Create a helper function stopDnAndWaitForNnToRemoveIt() for test cases to stop a DN and wait for it to be removed by DN.
          • Create a new MiniDFSCluster for every test case.

          Verified the testMetaSave runs fine on OSX & Linux (CentOS). Verified the run time reduced from 30+ seconds to 17 seconds.

          Show
          twu Tony Wu added a comment - In v2 patch: Use BlockManagerTestUtil#noticeDeadDatanode() to reduce the test run time. After this call the NN will declare the DN dead right away, rather than waiting for HB timeout. Create a helper function stopDnAndWaitForNnToRemoveIt() for test cases to stop a DN and wait for it to be removed by DN. Create a new MiniDFSCluster for every test case. Verified the testMetaSave runs fine on OSX & Linux (CentOS). Verified the run time reduced from 30+ seconds to 17 seconds.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for the prompt update.

          +1 (non-binding) pending on Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for the prompt update. +1 (non-binding) pending on Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 47s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_66
          +1 javac 0m 39s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.7.0_91
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
          -1 unit 51m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 50m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
          129m 37s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBpServiceActorScheduler
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.TestBlockReplacement



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777591/HDFS-9493.002.patch
          JIRA Issue HDFS-9493
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 814118cbff0b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1cb3299
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13871/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13871/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 47s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_91 +1 javac 0m 43s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 51m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 129m 37s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBpServiceActorScheduler   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777591/HDFS-9493.002.patch JIRA Issue HDFS-9493 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 814118cbff0b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1cb3299 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13871/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13871/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13871/console This message was automatically generated.
          Hide
          twu Tony Wu added a comment -

          The failed tests are not related to the patch. As the patch only updated TestMetaSave.java with a new helper function no other tests use.

          Show
          twu Tony Wu added a comment - The failed tests are not related to the patch. As the patch only updated TestMetaSave.java with a new helper function no other tests use.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Tony Wu

          Thanks a lot for addressing this failure. The overall idea looks very good. Will +1 once address the following comments

          • In isDatanodeRemoved, you don't need to hold FSNamesystem's read lock. NetworkTopology#contains holds its own read lock.
          • Would you mind to rename stopDnAndWaitForNnToRemoveIt() to something like stopDatanodeAndWait? Using Dn Nn is a little bit hard to read.
          • You might want to consider to set dfs.namenode.stale.datanode.interval to a smaller value.
          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Tony Wu Thanks a lot for addressing this failure. The overall idea looks very good. Will +1 once address the following comments In isDatanodeRemoved , you don't need to hold FSNamesystem 's read lock. NetworkTopology#contains holds its own read lock. Would you mind to rename stopDnAndWaitForNnToRemoveIt() to something like stopDatanodeAndWait ? Using Dn Nn is a little bit hard to read. You might want to consider to set dfs.namenode.stale.datanode.interval to a smaller value.
          Hide
          twu Tony Wu added a comment -

          Hi Lei (Eddy) Xu,

          Thanks a lot for reviewing my patch. I have incorporated your comments in the next patch. One note on dfs.namenode.stale.datanode.interval, I think reducing its value may not be necessary as BlockManagerTestUtil#noticeDeadDatanode() would make the NN "notice" the DN is dead immediately. I updated the parameter to be 5 seconds in case I'm missing something.

          Show
          twu Tony Wu added a comment - Hi Lei (Eddy) Xu , Thanks a lot for reviewing my patch. I have incorporated your comments in the next patch. One note on dfs.namenode.stale.datanode.interval , I think reducing its value may not be necessary as BlockManagerTestUtil#noticeDeadDatanode() would make the NN "notice" the DN is dead immediately. I updated the parameter to be 5 seconds in case I'm missing something.
          Hide
          twu Tony Wu added a comment -

          In v3 patch:

          • Addressed Lei (Eddy) Xu's review comments:
            • Dropped grabbing read lock for FSNamesystem
            • Renamed helper function in test to stopDatanodeAndWait()
            • Reduced dfs.namenode.stale.datanode.interval to 5 seconds
          Show
          twu Tony Wu added a comment - In v3 patch: Addressed Lei (Eddy) Xu 's review comments: Dropped grabbing read lock for FSNamesystem Renamed helper function in test to stopDatanodeAndWait() Reduced dfs.namenode.stale.datanode.interval to 5 seconds
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 22m 15s trunk passed
          +1 compile 3m 11s trunk passed with JDK v1.8.0_66
          +1 compile 2m 4s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 2m 39s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 26s trunk passed
          +1 javadoc 3m 23s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 21s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 55s the patch passed
          +1 compile 2m 41s the patch passed with JDK v1.8.0_66
          +1 javac 2m 41s the patch passed
          +1 compile 1m 45s the patch passed with JDK v1.7.0_91
          +1 javac 1m 45s the patch passed
          +1 checkstyle 0m 34s the patch passed
          +1 mvnsite 2m 16s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 42s the patch passed
          +1 javadoc 2m 59s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 14s the patch passed with JDK v1.7.0_91
          -1 unit 177m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 157m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 1m 26s Patch generated 1 ASF License warnings.
          410m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778636/HDFS-9493.003.patch
          JIRA Issue HDFS-9493
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6416fad69608 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9329564
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13944/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13944/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 22m 15s trunk passed +1 compile 3m 11s trunk passed with JDK v1.8.0_66 +1 compile 2m 4s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 40s trunk passed +1 mvnsite 2m 39s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 26s trunk passed +1 javadoc 3m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 21s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 55s the patch passed +1 compile 2m 41s the patch passed with JDK v1.8.0_66 +1 javac 2m 41s the patch passed +1 compile 1m 45s the patch passed with JDK v1.7.0_91 +1 javac 1m 45s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 2m 16s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 42s the patch passed +1 javadoc 2m 59s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 14s the patch passed with JDK v1.7.0_91 -1 unit 177m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 157m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 1m 26s Patch generated 1 ASF License warnings. 410m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778636/HDFS-9493.003.patch JIRA Issue HDFS-9493 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6416fad69608 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9329564 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13944/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13944/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13944/console This message was automatically generated.
          Hide
          twu Tony Wu added a comment -

          The test failures are not related to this patch as this patch only affects TestMetaSave, witch passes.

          Show
          twu Tony Wu added a comment - The test failures are not related to this patch as this patch only affects TestMetaSave, witch passes.
          Hide
          twu Tony Wu added a comment -

          Hi Lei (Eddy) Xu,

          Could you take a look at the latest patch (v3) and let me know if you have comments?

          Thanks,
          Tony

          Show
          twu Tony Wu added a comment - Hi Lei (Eddy) Xu , Could you take a look at the latest patch (v3) and let me know if you have comments? Thanks, Tony
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. Thanks Tony.

          Committed to trunk and branch-2

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. Thanks Tony. Committed to trunk and branch-2
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9075/)
          HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (lei: rev fd8065a763ff68db265ef23a7d4f97558e213ef9)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9075/ ) HDFS-9493 . Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (lei: rev fd8065a763ff68db265ef23a7d4f97558e213ef9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          twu Tony Wu added a comment -

          Thanks Mingliang Liu & Lei (Eddy) Xu for the review and comments!

          Show
          twu Tony Wu added a comment - Thanks Mingliang Liu & Lei (Eddy) Xu for the review and comments!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Lei (Eddy) Xu, there is a branch-2.8 where you need to land this patch for it to make to 2.8.0.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Lei (Eddy) Xu , there is a branch-2.8 where you need to land this patch for it to make to 2.8.0.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Vinod Kumar Vavilapalli Thanks for reminding me! Cherry picked it into branch-2.8 now.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Vinod Kumar Vavilapalli Thanks for reminding me! Cherry picked it into branch-2.8 now.

            People

            • Assignee:
              twu Tony Wu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development