Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9580

TestComputeInvalidateWork#testDatanodeReRegistration failed due to unexpected number of invalidate blocks.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, namenode, test
    • Labels:
      None
    • Environment:

      Jenkins

    • Hadoop Flags:
      Reviewed

      Description

      The failure appeared in the trunk jenkins job.
      https://builds.apache.org/job/Hadoop-Hdfs-trunk/2646/

      Error Message
      
      Expected invalidate blocks to be the number of DNs expected:<3> but was:<2>
      
      Stacktrace
      
      java.lang.AssertionError: Expected invalidate blocks to be the number of DNs expected:<3> but was:<2>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork.testDatanodeReRegistration(TestComputeInvalidateWork.java:160)
      
      

      I think there could be a race condition between creating a file and shutting down data nodes, which failed the test.

      2015-12-19 07:11:02,765 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=LAST_IN_PIPELINE, downstreams=0:[]] INFO  datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
      2015-12-19 07:11:02,768 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO  DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:45655, dest: /127.0.0.1:54890, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: 6a13ec05-e1c1-4086-8a4d-d5a09636afcd, blockid: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: 954174423
      2015-12-19 07:11:02,768 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO  datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
      2015-12-19 07:11:02,772 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO  DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:33252, dest: /127.0.0.1:54426, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: d81751db-02a9-48fe-b697-77623048784b, blockid: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: 957463510
      2015-12-19 07:11:02,772 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO  datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
      2015-12-19 07:11:02,782 [IPC Server handler 4 on 36404] INFO  blockmanagement.BlockManager (BlockManager.java:checkBlocksProperlyReplicated(3871)) - BLOCK* blk_1073741825_1001 is not COMPLETE (ucState = COMMITTED, replication# = 0 <  minimum = 1) in file /testRR
      2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO  namenode.EditLogFileOutputStream (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush
      2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO  namenode.EditLogFileOutputStream (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush
      2015-12-19 07:11:03,190 [IPC Server handler 8 on 36404] INFO  hdfs.StateChange (FSNamesystem.java:completeFile(2557)) - DIR* completeFile: /testRR is closed by DFSClient_NONMAPREDUCE_147911011_935
      
      1. HDFS-9580.001.patch
        0.9 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev01 patch. resolve race condition by ensuring the file is properly replicated before shutting down data nodes.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev01 patch. resolve race condition by ensuring the file is properly replicated before shutting down data nodes.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Wei-Chiu, good analysis. +1 on the patch pending Jenkins.

          Show
          zhz Zhe Zhang added a comment - Thanks Wei-Chiu, good analysis. +1 on the patch pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 46s trunk passed
          +1 compile 0m 44s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_66
          +1 javac 0m 42s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_91
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91
          -1 unit 63m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 32s Patch does not generate ASF License warnings.
          161m 6s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestReplication
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestSetTimes
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778739/HDFS-9580.001.patch
          JIRA Issue HDFS-9580
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4462174eef51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a0de702
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13963/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13963/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 46s trunk passed +1 compile 0m 44s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_91 +1 javac 0m 42s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91 -1 unit 63m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 32s Patch does not generate ASF License warnings. 161m 6s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestReplication JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestSetTimes   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778739/HDFS-9580.001.patch JIRA Issue HDFS-9580 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4462174eef51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0de702 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13963/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13963/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Test failures look unrelated.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Test failures look unrelated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Wei-Chiu. I just committed the patch to trunk.

          Show
          zhz Zhe Zhang added a comment - Thanks Wei-Chiu. I just committed the patch to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9011 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9011/)
          HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed (zhz: rev 0087734cc1e18e029ea39f06ca25c8a7d60af004)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9011 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9011/ ) HDFS-9580 . TestComputeInvalidateWork#testDatanodeReRegistration failed (zhz: rev 0087734cc1e18e029ea39f06ca25c8a7d60af004) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thank you Zhe Zhang for the review and the commit!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thank you Zhe Zhang for the review and the commit!
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Wei-Chiu Chuang! I committed this to branch-2 and branch-2.8 as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Wei-Chiu Chuang ! I committed this to branch-2 and branch-2.8 as well.

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development