Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8224

Schedule a block for scanning if its metadata file is corrupt

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This happened in our 2.6 cluster.
      One of the block and its metadata file were corrupted.
      The disk was healthy in this case.
      Only the block was corrupt.

      Namenode tried to copy that block to another datanode but failed with the following stack trace:

      2015-04-20 01:04:04,421 [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN datanode.DataNode: DatanodeRegistration(a.b.c.d, datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006, infoSecurePort=0, ipcPort=8020, storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to a1.b1.c1.d1:1004 got
      java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0
      at org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
      at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
      at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
      at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:287)
      at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
      at java.lang.Thread.run(Thread.java:722)

      The following catch block in DataTransfer#run method will treat every IOException as disk error fault and run disk errror

      catch (IOException ie) {
              LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
                  targets[0] + " got ", ie);
              // check if there are any disk problem
              checkDiskErrorAsync();
            } 
      

      This block was never scanned by BlockPoolSliceScanner otherwise it would have reported as corrupt block.

      1. HDFS-8224-branch-2.7.patch
        9 kB
        Wei-Chiu Chuang
      2. HDFS-8224-branch-2.patch
        9 kB
        Rushabh S Shah
      3. HDFS-8224-trunk.patch
        11 kB
        Rushabh S Shah
      4. HDFS-8224-trunk-1.patch
        9 kB
        Rushabh S Shah
      5. HDFS-8224-trunk-2.patch
        11 kB
        Rushabh S Shah
      6. HDFS-8224-trunk-3.patch
        9 kB
        Rushabh S Shah

        Activity

        Hide
        kihwal Kihwal Lee added a comment -

        Sometimes it is unclear whether the cause of a failure is from the network or the local io.
        Since we have eliminated unnecessary recursive checks in checkDiskError, there is almost no penalty in more aggressively doing checkDisk.

        Show
        kihwal Kihwal Lee added a comment - Sometimes it is unclear whether the cause of a failure is from the network or the local io. Since we have eliminated unnecessary recursive checks in checkDiskError, there is almost no penalty in more aggressively doing checkDisk.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Saw one more occurrence of this bug.
        We should atleast add this block to the front of the scanning queue.

        Show
        shahrs87 Rushabh S Shah added a comment - Saw one more occurrence of this bug. We should atleast add this block to the front of the scanning queue.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Rushabh S Shah thanks for bring up this discussion. What would be the relation between this proposal versus HDFS-10627? In its current form, if blockSender.sendPacket() gets a Connection Reset or Broken pipe, the block is added to the scanning queue of VolumeScanner. So that would mean the block is scanned twice if this case happens. Or we could move that piece of code to the catch block of DataTransfer#run() to add the block into scanning queue there.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Rushabh S Shah thanks for bring up this discussion. What would be the relation between this proposal versus HDFS-10627 ? In its current form, if blockSender.sendPacket() gets a Connection Reset or Broken pipe, the block is added to the scanning queue of VolumeScanner. So that would mean the block is scanned twice if this case happens. Or we could move that piece of code to the catch block of DataTransfer#run() to add the block into scanning queue there.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        The exception in this jira is occurring at BlockSender constructor.

          blockSender = new BlockSender(b, 0, b.getNumBytes(), 
                    false, false, true, DataNode.this, null, cachingStrategy);
        

        The exception mentioned in HDFS-10627 is occurring at:

         // send data & checksum
                blockSender.sendBlock(out, unbufOut, null);
        

        For this jira, I was thinking as follows:

        DataChecksum.java
        public static DataChecksum newDataChecksum( DataInputStream in )
                                         throws IOException {
            int type = in.readByte();
            int bpc = in.readInt();
            DataChecksum summer = newDataChecksum(Type.valueOf(type), bpc );
            if ( summer == null ) {
              throw new IOException( "Could not create DataChecksum of type " +
                                     type + " with bytesPerChecksum " + bpc );
            }
            return summer;
          }
        

        If we can throw TypeZeroException instead of IOException (which ofcourse extends IOException) in case if summer == null
        Since summer will be null only if bytesPerChecksum <= 0

        DataChecksum.java
         public static DataChecksum newDataChecksum(Type type, int bytesPerChecksum ) {
            if ( bytesPerChecksum <= 0 ) {
              return null;
            }
            
            switch ( type ) {
            case NULL :
              return new DataChecksum(type, new ChecksumNull(), bytesPerChecksum );
            case CRC32 :
              return new DataChecksum(type, newCrc32(), bytesPerChecksum );
            case CRC32C:
              return new DataChecksum(type, new PureJavaCrc32C(), bytesPerChecksum);
            default:
              return null;  
            }
          }
        

        In the DataTransfer#run method, either we can add a try block across BlockSender constructor and check if thrown exception is an instance of TypeZeroException or in the catch block as per the code today.
        If it is TypeZeroException then we can add it to scanning queue and keep the remaining logic as it is
        Wei-Chiu Chuang: Any thoughts ?

        Show
        shahrs87 Rushabh S Shah added a comment - The exception in this jira is occurring at BlockSender constructor. blockSender = new BlockSender(b, 0, b.getNumBytes(), false, false, true, DataNode.this, null, cachingStrategy); The exception mentioned in HDFS-10627 is occurring at: // send data & checksum blockSender.sendBlock(out, unbufOut, null); For this jira, I was thinking as follows: DataChecksum.java public static DataChecksum newDataChecksum( DataInputStream in ) throws IOException { int type = in.readByte(); int bpc = in.readInt(); DataChecksum summer = newDataChecksum(Type.valueOf(type), bpc ); if ( summer == null ) { throw new IOException( "Could not create DataChecksum of type " + type + " with bytesPerChecksum " + bpc ); } return summer; } If we can throw TypeZeroException instead of IOException (which ofcourse extends IOException) in case if summer == null Since summer will be null only if bytesPerChecksum <= 0 DataChecksum.java public static DataChecksum newDataChecksum(Type type, int bytesPerChecksum ) { if ( bytesPerChecksum <= 0 ) { return null ; } switch ( type ) { case NULL : return new DataChecksum(type, new ChecksumNull(), bytesPerChecksum ); case CRC32 : return new DataChecksum(type, newCrc32(), bytesPerChecksum ); case CRC32C: return new DataChecksum(type, new PureJavaCrc32C(), bytesPerChecksum); default : return null ; } } In the DataTransfer#run method, either we can add a try block across BlockSender constructor and check if thrown exception is an instance of TypeZeroException or in the catch block as per the code today. If it is TypeZeroException then we can add it to scanning queue and keep the remaining logic as it is Wei-Chiu Chuang : Any thoughts ?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Incidentally, we just saw the same stack trace "java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0" in a cluster. I wonder if it's caused by a software (HDFS) bug rather than a hardware failure.

        Based on my understanding, the stacktrace suggests it is the meta file of the replica that's corrupt (its header has invalid values: type 0 and bytesPerChecksum 0). It might be possible the replica itself is actually good.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Incidentally, we just saw the same stack trace "java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0" in a cluster. I wonder if it's caused by a software (HDFS) bug rather than a hardware failure. Based on my understanding, the stacktrace suggests it is the meta file of the replica that's corrupt (its header has invalid values: type 0 and bytesPerChecksum 0). It might be possible the replica itself is actually good.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rushabh S Shah I think that looks generally good to me. I only have one comment: The null thrown by DataChecksum#newDataChecksum is caused by zero bytesPerChecksum. So instead of throwing a TypeZeroException you might want to throw a InvalidChecksumSizeException or something like that.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rushabh S Shah I think that looks generally good to me. I only have one comment: The null thrown by DataChecksum#newDataChecksum is caused by zero bytesPerChecksum. So instead of throwing a TypeZeroException you might want to throw a InvalidChecksumSizeException or something like that.
        Hide
        kihwal Kihwal Lee added a comment -

        Incidentally, we just saw the same stack trace "java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0" in a cluster. I wonder if it's caused by a software (HDFS) bug rather than a hardware failure.

        We have seen this type of errors caused by flaky disks. The kernel(ext4) would say delayed block allocations failed and part of the block and/or meta file ends up being all null. This happens asynchronous to user's I/O activities, so the writes don't get any errors. Unfortunately including data_err=abort,errors=remount-ro in the mount option does not make it go read-only right away.

        Show
        kihwal Kihwal Lee added a comment - Incidentally, we just saw the same stack trace "java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0" in a cluster. I wonder if it's caused by a software (HDFS) bug rather than a hardware failure. We have seen this type of errors caused by flaky disks. The kernel(ext4) would say delayed block allocations failed and part of the block and/or meta file ends up being all null. This happens asynchronous to user's I/O activities, so the writes don't get any errors. Unfortunately including data_err=abort,errors=remount-ro in the mount option does not make it go read-only right away.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Attaching a patch.
        please review.

        Show
        shahrs87 Rushabh S Shah added a comment - Attaching a patch. please review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 8m 29s trunk passed
        +1 compile 9m 24s trunk passed
        +1 checkstyle 1m 30s trunk passed
        +1 mvnsite 1m 54s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 3m 9s trunk passed
        +1 javadoc 1m 52s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 35s the patch passed
        +1 compile 7m 11s the patch passed
        +1 javac 7m 11s the patch passed
        -0 checkstyle 1m 29s root: The patch generated 4 new + 497 unchanged - 2 fixed = 501 total (was 499)
        +1 mvnsite 1m 45s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 23s the patch passed
        +1 javadoc 1m 51s the patch passed
        -1 unit 19m 47s hadoop-common in the patch failed.
        -1 unit 79m 17s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        146m 22s



        Reason Tests
        Failed junit tests hadoop.tracing.TestTracing
          hadoop.hdfs.TestRollingUpgrade
          hadoop.security.TestRefreshUserMappings
        Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822176/HDFS-8224-trunk.patch
        JIRA Issue HDFS-8224
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4e3bb71e3e51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 438a9f0
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16324/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16324/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 29s trunk passed +1 compile 9m 24s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 52s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 7m 11s the patch passed +1 javac 7m 11s the patch passed -0 checkstyle 1m 29s root: The patch generated 4 new + 497 unchanged - 2 fixed = 501 total (was 499) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 23s the patch passed +1 javadoc 1m 51s the patch passed -1 unit 19m 47s hadoop-common in the patch failed. -1 unit 79m 17s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 146m 22s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.TestRollingUpgrade   hadoop.security.TestRefreshUserMappings Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822176/HDFS-8224-trunk.patch JIRA Issue HDFS-8224 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4e3bb71e3e51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 438a9f0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16324/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16324/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16324/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Cancelling the patch to address checkstyle issues.

        Following are the failing tests:
        TestRollingUpgrade#testRollback is failing and is tracked via HDFS-9664
        TestTracing#testTracing is failing and tracked via HADOOP-13473
        TestHttpServerLifecycle is running fine on my machine.
        TestRefreshUserMappings is failing with and without my patch.
        Will open a ticket to track this.

        Show
        shahrs87 Rushabh S Shah added a comment - Cancelling the patch to address checkstyle issues. Following are the failing tests: TestRollingUpgrade#testRollback is failing and is tracked via HDFS-9664 TestTracing#testTracing is failing and tracked via HADOOP-13473 TestHttpServerLifecycle is running fine on my machine. TestRefreshUserMappings is failing with and without my patch. Will open a ticket to track this.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Wei-Chiu Chuang: Since you know the most context, do you mind reviewing the patch ?

        Show
        shahrs87 Rushabh S Shah added a comment - Wei-Chiu Chuang : Since you know the most context, do you mind reviewing the patch ?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Sure I'll review soon. Thx for the patch.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Sure I'll review soon. Thx for the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 33s trunk passed
        +1 compile 6m 39s trunk passed
        +1 checkstyle 1m 31s trunk passed
        +1 mvnsite 1m 44s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 2m 55s trunk passed
        +1 javadoc 1m 40s trunk passed
        0 mvndep 0m 13s Maven dependency ordering for patch
        -1 mvninstall 0m 19s hadoop-common in the patch failed.
        -1 mvninstall 0m 24s hadoop-hdfs in the patch failed.
        -1 compile 0m 31s root in the patch failed.
        -1 javac 0m 31s root in the patch failed.
        +1 checkstyle 1m 26s root: The patch generated 0 new + 497 unchanged - 2 fixed = 497 total (was 499)
        -1 mvnsite 0m 21s hadoop-common in the patch failed.
        -1 mvnsite 0m 25s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 18s hadoop-common in the patch failed.
        -1 findbugs 0m 24s hadoop-hdfs in the patch failed.
        -1 javadoc 0m 58s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
        -1 unit 0m 21s hadoop-common in the patch failed.
        -1 unit 0m 26s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        29m 56s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822831/HDFS-8224-trunk-1.patch
        JIRA Issue HDFS-8224
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d0d3f2244de6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b10c936
        Default Java 1.8.0_101
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-compile-root.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16361/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16361/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 33s trunk passed +1 compile 6m 39s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 55s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch -1 mvninstall 0m 19s hadoop-common in the patch failed. -1 mvninstall 0m 24s hadoop-hdfs in the patch failed. -1 compile 0m 31s root in the patch failed. -1 javac 0m 31s root in the patch failed. +1 checkstyle 1m 26s root: The patch generated 0 new + 497 unchanged - 2 fixed = 497 total (was 499) -1 mvnsite 0m 21s hadoop-common in the patch failed. -1 mvnsite 0m 25s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 18s hadoop-common in the patch failed. -1 findbugs 0m 24s hadoop-hdfs in the patch failed. -1 javadoc 0m 58s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) -1 unit 0m 21s hadoop-common in the patch failed. -1 unit 0m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 29m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822831/HDFS-8224-trunk-1.patch JIRA Issue HDFS-8224 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d0d3f2244de6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b10c936 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-compile-root.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16361/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16361/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16361/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Cancelling the patch since forgot to add a new file in the patch.

        Show
        shahrs87 Rushabh S Shah added a comment - Cancelling the patch since forgot to add a new file in the patch.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Attaching a patch for trunk.
        Will post a patch for branch-2.8 and branch-2.9 once trunk one get reviewed.

        Show
        shahrs87 Rushabh S Shah added a comment - Attaching a patch for trunk. Will post a patch for branch-2.8 and branch-2.9 once trunk one get reviewed.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        TestRefreshUserMappings is failing with and without my patch.
        Will open a ticket to track this.

        Filed HDFS-10741 to track this.

        Show
        shahrs87 Rushabh S Shah added a comment - TestRefreshUserMappings is failing with and without my patch. Will open a ticket to track this. Filed HDFS-10741 to track this.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the patch! Overall it looks good to me. I just like to add a few nits:

        • Can you move the test to TestDiskError.java? In this way, you do not need to make BlockScanner#markSuspectBlock, DataNode#setBlockScanner and DataNode#transferBlock public. I also think this is more nature to place this test in this test class.
        • InvalidChecksumSizeException.java
           * Thrown when bytesPerChecksun field in the meta file is less than
           * or equal to 0.
          

          To be more precise, the exception can also be thrown if the type is invalid.

        • The following line should be removed.
          TestDataTransferProtocol.java
          //config.setLong(DFS_DATANODE_SCAN_PERIOD_HOURS_KEY, -1);
          
        • Finally, could you add a comment here that basically says if the peer disconnects the block is already added to BlockScanner, so do not add to the scan queue again. However an InvalidChecksumSizeException is thrown because metafile is corrupt (caused by a flaky disk) and therefore add to scan queue here.
          DataNode.java
                } catch (IOException ie) {
                  if (ie instanceof InvalidChecksumSizeException) {
           
          
        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the patch! Overall it looks good to me. I just like to add a few nits: Can you move the test to TestDiskError.java? In this way, you do not need to make BlockScanner#markSuspectBlock , DataNode#setBlockScanner and DataNode#transferBlock public. I also think this is more nature to place this test in this test class. InvalidChecksumSizeException.java * Thrown when bytesPerChecksun field in the meta file is less than * or equal to 0. To be more precise, the exception can also be thrown if the type is invalid. The following line should be removed. TestDataTransferProtocol.java //config.setLong(DFS_DATANODE_SCAN_PERIOD_HOURS_KEY, -1); Finally, could you add a comment here that basically says if the peer disconnects the block is already added to BlockScanner, so do not add to the scan queue again. However an InvalidChecksumSizeException is thrown because metafile is corrupt (caused by a flaky disk) and therefore add to scan queue here. DataNode.java } catch (IOException ie) { if (ie instanceof InvalidChecksumSizeException) {
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 7m 28s trunk passed
        +1 compile 7m 11s trunk passed
        +1 checkstyle 1m 29s trunk passed
        +1 mvnsite 1m 54s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 6s trunk passed
        +1 javadoc 1m 42s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 7m 3s the patch passed
        +1 javac 7m 3s the patch passed
        +1 checkstyle 1m 30s root: The patch generated 0 new + 497 unchanged - 2 fixed = 497 total (was 499)
        +1 mvnsite 1m 52s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 24s the patch passed
        +1 javadoc 1m 42s the patch passed
        +1 unit 8m 12s hadoop-common in the patch passed.
        -1 unit 71m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        122m 32s



        Reason Tests
        Failed junit tests hadoop.tracing.TestTracing
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.security.TestRefreshUserMappings



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822850/HDFS-8224-trunk-2.patch
        JIRA Issue HDFS-8224
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 59cb6e669104 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c4b77ae
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16365/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16365/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16365/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 28s trunk passed +1 compile 7m 11s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 7m 3s the patch passed +1 javac 7m 3s the patch passed +1 checkstyle 1m 30s root: The patch generated 0 new + 497 unchanged - 2 fixed = 497 total (was 499) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 42s the patch passed +1 unit 8m 12s hadoop-common in the patch passed. -1 unit 71m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 122m 32s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.security.TestRefreshUserMappings Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822850/HDFS-8224-trunk-2.patch JIRA Issue HDFS-8224 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 59cb6e669104 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c4b77ae Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16365/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16365/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16365/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Wei-Chiu Chuang: Thanks a lot for your valuable reviews.
        Cancelling the patch to address your comments.

        Show
        shahrs87 Rushabh S Shah added a comment - Wei-Chiu Chuang : Thanks a lot for your valuable reviews. Cancelling the patch to address your comments.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Created a new patch addressing all the previous comments.
        Wei-Chiu Chuang: please review.

        Show
        shahrs87 Rushabh S Shah added a comment - Created a new patch addressing all the previous comments. Wei-Chiu Chuang : please review.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        +1 pending Jenkins.

        Show
        jojochuang Wei-Chiu Chuang added a comment - +1 pending Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 30s trunk passed
        +1 compile 7m 4s trunk passed
        +1 checkstyle 1m 30s trunk passed
        +1 mvnsite 1m 52s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 9s trunk passed
        +1 javadoc 1m 42s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 7m 21s the patch passed
        +1 javac 7m 21s the patch passed
        +1 checkstyle 1m 27s root: The patch generated 0 new + 310 unchanged - 2 fixed = 310 total (was 312)
        +1 mvnsite 1m 50s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 3m 30s the patch passed
        +1 javadoc 1m 39s the patch passed
        +1 unit 7m 21s hadoop-common in the patch passed.
        -1 unit 64m 57s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        114m 5s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822898/HDFS-8224-trunk-3.patch
        JIRA Issue HDFS-8224
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0c910e4273b6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 85422bb
        Default Java 1.8.0_101
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16371/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16371/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16371/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16371/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 30s trunk passed +1 compile 7m 4s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 7m 21s the patch passed +1 javac 7m 21s the patch passed +1 checkstyle 1m 27s root: The patch generated 0 new + 310 unchanged - 2 fixed = 310 total (was 312) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 30s the patch passed +1 javadoc 1m 39s the patch passed +1 unit 7m 21s hadoop-common in the patch passed. -1 unit 64m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 114m 5s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822898/HDFS-8224-trunk-3.patch JIRA Issue HDFS-8224 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c910e4273b6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 85422bb Default Java 1.8.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16371/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16371/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16371/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16371/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The test failures are unrelated. Committing the v3 patch.

        Show
        jojochuang Wei-Chiu Chuang added a comment - The test failures are unrelated. Committing the v3 patch.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed to trunk. Thanks a lot Rushabh S Shah for the patch and Kihwal Lee for comments.
        Can you also upload a branch-2 patch? There are conflicts cherrypicking from trunk.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed to trunk. Thanks a lot Rushabh S Shah for the patch and Kihwal Lee for comments. Can you also upload a branch-2 patch? There are conflicts cherrypicking from trunk.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10251/)
        HDFS-8224. Schedule a block for scanning if its metadata file is (weichiu: rev d00d3add9e3c7ac7e79bb99b615bcfaeed892b96)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InvalidChecksumSizeException.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10251/ ) HDFS-8224 . Schedule a block for scanning if its metadata file is (weichiu: rev d00d3add9e3c7ac7e79bb99b615bcfaeed892b96) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InvalidChecksumSizeException.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Wei-Chiu Chuang: Thanks a lot for the reviews and committing.

        Show
        shahrs87 Rushabh S Shah added a comment - Wei-Chiu Chuang : Thanks a lot for the reviews and committing.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Attaching a branch-2 patch.
        Same as trunk. Just fixed one whitespace warning.

        Show
        shahrs87 Rushabh S Shah added a comment - Attaching a branch-2 patch. Same as trunk. Just fixed one whitespace warning.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 57s Maven dependency ordering for branch
        +1 mvninstall 6m 54s branch-2 passed
        +1 compile 6m 6s branch-2 passed with JDK v1.8.0_101
        +1 compile 6m 37s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 1m 30s branch-2 passed
        +1 mvnsite 1m 49s branch-2 passed
        +1 mvneclipse 0m 31s branch-2 passed
        +1 findbugs 3m 33s branch-2 passed
        +1 javadoc 1m 43s branch-2 passed with JDK v1.8.0_101
        +1 javadoc 2m 32s branch-2 passed with JDK v1.7.0_101
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 5m 54s the patch passed with JDK v1.8.0_101
        +1 javac 5m 54s the patch passed
        +1 compile 6m 30s the patch passed with JDK v1.7.0_101
        +1 javac 6m 30s the patch passed
        +1 checkstyle 1m 24s root: The patch generated 0 new + 311 unchanged - 2 fixed = 311 total (was 313)
        +1 mvnsite 1m 50s the patch passed
        +1 mvneclipse 0m 30s the patch passed
        -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 4m 2s the patch passed
        +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101
        +1 javadoc 2m 33s the patch passed with JDK v1.7.0_101
        +1 unit 8m 40s hadoop-common in the patch passed with JDK v1.7.0_101.
        -1 unit 50m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        182m 4s



        Reason Tests
        JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823036/HDFS-8224-branch-2.patch
        JIRA Issue HDFS-8224
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3b7c58ae4b72 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 447ba47
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16381/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16381/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16381/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 57s Maven dependency ordering for branch +1 mvninstall 6m 54s branch-2 passed +1 compile 6m 6s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 37s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 1m 49s branch-2 passed +1 mvneclipse 0m 31s branch-2 passed +1 findbugs 3m 33s branch-2 passed +1 javadoc 1m 43s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 5m 54s the patch passed with JDK v1.8.0_101 +1 javac 5m 54s the patch passed +1 compile 6m 30s the patch passed with JDK v1.7.0_101 +1 javac 6m 30s the patch passed +1 checkstyle 1m 24s root: The patch generated 0 new + 311 unchanged - 2 fixed = 311 total (was 313) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 30s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 4m 2s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 33s the patch passed with JDK v1.7.0_101 +1 unit 8m 40s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 50m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 182m 4s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823036/HDFS-8224-branch-2.patch JIRA Issue HDFS-8224 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3b7c58ae4b72 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 447ba47 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16381/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16381/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16381/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed it to trunk, branch-2 and branch-2.8.

        Thanks again Rushabh S Shah!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed it to trunk, branch-2 and branch-2.8. Thanks again Rushabh S Shah !
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Do we want it to commit it to 2.7.4 ?

        Show
        shahrs87 Rushabh S Shah added a comment - Do we want it to commit it to 2.7.4 ?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Yeah given this error could potentially impair HDFS's ability to maintain sufficient number of good replicas, I think it makes sense to commit to 2.7.x or even 2.6.x.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Yeah given this error could potentially impair HDFS's ability to maintain sufficient number of good replicas, I think it makes sense to commit to 2.7.x or even 2.6.x.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Attached a branch-2.7 patch. There was a small import conflict.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Attached a branch-2.7 patch. There was a small import conflict.
        Hide
        shahrs87 Rushabh S Shah added a comment - - edited

        +1 lgtm (non-binding) for 2.7 patch.
        Wei-Chiu Chuang: Thanks for uploading the patch.

        Show
        shahrs87 Rushabh S Shah added a comment - - edited +1 lgtm (non-binding) for 2.7 patch. Wei-Chiu Chuang : Thanks for uploading the patch.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed this to 2.7.
        Branch 2.6 uses the old implementation of DataBlockScanner, so this patch is not compatible.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed this to 2.7. Branch 2.6 uses the old implementation of DataBlockScanner, so this patch is not compatible.

          People

          • Assignee:
            shahrs87 Rushabh S Shah
            Reporter:
            shahrs87 Rushabh S Shah
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development