Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8498

Blocks can be committed with wrong size

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When an IBR for a UC block arrives, the NN updates the expected location's block and replica state only if it's on an unexpected storage for an expected DN. If it's for an expected storage, only the genstamp is updated. When the block is committed, and the expected locations are verified, only the genstamp is checked. The size is not checked but it wasn't updated in the expected locations anyway.

      A faulty client may misreport the size when committing the block. The block is effectively corrupted. If the NN issues replications, the received IBR is considered corrupt, the NN invalidates the block, immediately issues another replication. The NN eventually realizes all the original replicas are corrupt after full BRs are received from the original DNs.

      1. HDFS-8498.000.patch
        10 kB
        Jing Zhao
      2. HDFS-8498.001.patch
        11 kB
        Jing Zhao
      3. HDFS-8498.branch-2.001.patch
        13 kB
        Jing Zhao
      4. HDFS-8498.branch-2.7.001.patch
        11 kB
        Wei-Chiu Chuang
      5. HDFS-8498.branch-2.patch
        13 kB
        Jing Zhao
      6. HDFS-8498-branch-2.7.002.patch
        10 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Daryn Sharp, is this a regression in 2.7.0?

          Do we have any proposals that we can get in shortly for 2.7.1. Otherwise, we should move this out. Please comment, tx.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp , is this a regression in 2.7.0? Do we have any proposals that we can get in shortly for 2.7.1. Otherwise, we should move this out. Please comment, tx.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Sounds like a critical issue to me, but got no response. Moving this to 2.7.2.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Sounds like a critical issue to me, but got no response. Moving this to 2.7.2.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Daryn Sharp, any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp , any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.
          Hide
          kihwal Kihwal Lee added a comment -

          This is an existing bug, I believe. Re-targeting to 2.7.3.

          Show
          kihwal Kihwal Lee added a comment - This is an existing bug, I believe. Re-targeting to 2.7.3.
          Hide
          zhz Zhe Zhang added a comment -

          A faulty client may misreport the size when committing the block.

          Daryn Sharp Do you have more details regarding the above client bug?

          We are having a somewhat related discussion on HDFS-9289.

          Show
          zhz Zhe Zhang added a comment - A faulty client may misreport the size when committing the block. Daryn Sharp Do you have more details regarding the above client bug? We are having a somewhat related discussion on HDFS-9289 .
          Hide
          daryn Daryn Sharp added a comment -

          I don't have any more details than in the description. We've seen the bug only once (that we know of). I looked at the code and described the bug I saw in it.

          Show
          daryn Daryn Sharp added a comment - I don't have any more details than in the description. We've seen the bug only once (that we know of). I looked at the code and described the bug I saw in it.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Zhe Zhang / Kihwal Lee / Daryn Sharp, I think we should close this as Won't fix or as a dup of HDFS-9289.

          This bug keeps appearing in the blocker/critical list for releases, but we don't seem to be progressing.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Zhe Zhang / Kihwal Lee / Daryn Sharp , I think we should close this as Won't fix or as a dup of HDFS-9289 . This bug keeps appearing in the blocker/critical list for releases, but we don't seem to be progressing.
          Hide
          zhz Zhe Zhang added a comment -

          I'm OK with closing as won't fix, if Daryn hasn't seen new occurrences of the bug after HDFS-9289.

          Show
          zhz Zhe Zhang added a comment - I'm OK with closing as won't fix, if Daryn hasn't seen new occurrences of the bug after HDFS-9289 .
          Hide
          kihwal Kihwal Lee added a comment -

          resolving, won't fix.

          Show
          kihwal Kihwal Lee added a comment - resolving, won't fix.
          Hide
          vinayrpet Vinayakumar B added a comment -

          We experienced this in one of our testing cluster under high load.

          Scenario:
          Error occured for the HBase-RegionServer's WAL file.
          1. In HBase there will be multiple threads performing the write,sync and close of same WAL file.
          2. Actual writer writes the entries, multiple syncers call hsync on same stream and A roller thread rolls the WALs in regular intervals. i.e. close the curren WAL file and open another one for next entries.
          3. During file close() by roller, last block got committed with less size, than present in all DNs.
          4. All IBRs reported by DNs have more length, than that of COMMITTED length by the client. So all those replicas are marked as CORRUPT.
          5. We use IBR batch with dfs.namenode.file.close.num-committed-allowed=1. So Client(HBase-RS) did not experience any problem, as file got closed successfully without waiting for the Correct IBR for last block.

          Current Analysis:

          HDFS-9289, safegaurded the DataStreamer#block's re-assignment during pipeline update by making it volatile. But it did not actually protected the contents of the block.

          Suspected problem is:
          1. ResponseProcessor updated the block size by updating the numBytes after receiving every Ack by calling ExtendedBlock.setNumBytes(), which internally updates the numBytes of internal block which is not thread safe.
          2. LogRoller calls close by by passing DataStreamer#block as last block. During this time, GUESS is that ExtendedBlock.getNumBytes() is not returning the latest value updated by ReponseProcessor, instead returning some of the earlier update. Because ExtendedBlock and its internal block is not threadsafe.
          By this lesser size, Block is getting COMMITTED at NameNode and all IBRs are getting marked as CORRUPT.

          Possible solution:
          Make the ExtendedBlock threadsafe for setNumBytes() and getNumBytes().

          If the above analysis makes sense, then we can raise one Jira and contribute the fix.

          Note:
          This issue we got in 40-core/380GB-RAM machine thrice. Trying to reproduce again with more logs, but no luck till now.
          Once it was reproduced with DEBUG logs as well, from that its confirmed that complete() call is sent only after receiving all ACKs. But DEBUG logs was having no information of numBytes sent during complete(). So could not actually verify that this would be the fix.

          Show
          vinayrpet Vinayakumar B added a comment - We experienced this in one of our testing cluster under high load. Scenario: Error occured for the HBase-RegionServer's WAL file. 1. In HBase there will be multiple threads performing the write,sync and close of same WAL file. 2. Actual writer writes the entries, multiple syncers call hsync on same stream and A roller thread rolls the WALs in regular intervals. i.e. close the curren WAL file and open another one for next entries. 3. During file close() by roller, last block got committed with less size, than present in all DNs. 4. All IBRs reported by DNs have more length, than that of COMMITTED length by the client. So all those replicas are marked as CORRUPT. 5. We use IBR batch with dfs.namenode.file.close.num-committed-allowed=1 . So Client(HBase-RS) did not experience any problem, as file got closed successfully without waiting for the Correct IBR for last block. Current Analysis: HDFS-9289 , safegaurded the DataStreamer#block 's re-assignment during pipeline update by making it volatile. But it did not actually protected the contents of the block . Suspected problem is: 1. ResponseProcessor updated the block size by updating the numBytes after receiving every Ack by calling ExtendedBlock.setNumBytes() , which internally updates the numBytes of internal block which is not thread safe. 2. LogRoller calls close by by passing DataStreamer#block as last block. During this time, GUESS is that ExtendedBlock.getNumBytes() is not returning the latest value updated by ReponseProcessor, instead returning some of the earlier update. Because ExtendedBlock and its internal block is not threadsafe. By this lesser size, Block is getting COMMITTED at NameNode and all IBRs are getting marked as CORRUPT. Possible solution: Make the ExtendedBlock threadsafe for setNumBytes() and getNumBytes(). If the above analysis makes sense, then we can raise one Jira and contribute the fix. Note: This issue we got in 40-core/380GB-RAM machine thrice. Trying to reproduce again with more logs, but no luck till now. Once it was reproduced with DEBUG logs as well, from that its confirmed that complete() call is sent only after receiving all ACKs. But DEBUG logs was having no information of numBytes sent during complete(). So could not actually verify that this would be the fix.
          Hide
          jingzhao Jing Zhao added a comment -

          We also saw the same scenario as described by Vinayakumar B. Maybe we can reopen this jira and explore the solution proposed by Vinayakumar B.

          Show
          jingzhao Jing Zhao added a comment - We also saw the same scenario as described by Vinayakumar B . Maybe we can reopen this jira and explore the solution proposed by Vinayakumar B .
          Hide
          jingzhao Jing Zhao added a comment -

          Vinay's proposed solution looks good to me. For the implementation, instead of directly changing ExtendedBlock which is used everywhere, maybe we can create a Block-similar structure which is thread safe and only used by DFSOutputStream/DataStreamer internally.

          Upload a patch to demo the idea. Please comment.

          Show
          jingzhao Jing Zhao added a comment - Vinay's proposed solution looks good to me. For the implementation, instead of directly changing ExtendedBlock which is used everywhere, maybe we can create a Block-similar structure which is thread safe and only used by DFSOutputStream/DataStreamer internally. Upload a patch to demo the idea. Please comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 24m 25s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 33s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 27s the patch passed
          +1 javac 0m 27s the patch passed
          -0 checkstyle 0m 13s hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 3 new + 77 unchanged - 2 fixed = 80 total (was 79)
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 30s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 17s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          33m 59s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
            Non-virtual method call in new org.apache.hadoop.hdfs.StripedDataStreamer(HdfsFileStatus, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], short, DFSStripedOutputStream$Coordinator, EnumSet) passes null for non-null parameter of new DataStreamer(HdfsFileStatus, ExtendedBlock, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], EnumSet) At StripedDataStreamer.java:org.apache.hadoop.hdfs.StripedDataStreamer(HdfsFileStatus, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], short, DFSStripedOutputStream$Coordinator, EnumSet) passes null for non-null parameter of new DataStreamer(HdfsFileStatus, ExtendedBlock, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], EnumSet) At StripedDataStreamer.java:[line 58]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847960/HDFS-8498.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 79a18db5ce92 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / efc8faa
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18212/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18212/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18212/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18212/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 24m 25s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed -0 checkstyle 0m 13s hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 3 new + 77 unchanged - 2 fixed = 80 total (was 79) +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 30s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 17s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 33m 59s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   Non-virtual method call in new org.apache.hadoop.hdfs.StripedDataStreamer(HdfsFileStatus, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], short, DFSStripedOutputStream$Coordinator, EnumSet) passes null for non-null parameter of new DataStreamer(HdfsFileStatus, ExtendedBlock, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], EnumSet) At StripedDataStreamer.java:org.apache.hadoop.hdfs.StripedDataStreamer(HdfsFileStatus, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], short, DFSStripedOutputStream$Coordinator, EnumSet) passes null for non-null parameter of new DataStreamer(HdfsFileStatus, ExtendedBlock, DFSClient, String, Progressable, DataChecksum, AtomicReference, ByteArrayManager, String[], EnumSet) At StripedDataStreamer.java: [line 58] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847960/HDFS-8498.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 79a18db5ce92 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / efc8faa Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18212/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18212/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18212/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18212/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Update the patch to fix bug when block is initialized as null. Also slightly changed TestDFSOutputStream.java to trigger tests in hadoop-hdfs.

          Show
          jingzhao Jing Zhao added a comment - Update the patch to fix bug when block is initialized as null. Also slightly changed TestDFSOutputStream.java to trigger tests in hadoop-hdfs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 11s trunk passed
          +1 compile 1m 26s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 17s trunk passed
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 1m 24s the patch passed
          +1 javac 1m 24s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 33s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 90m 9s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          124m 8s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf38032a353b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5839fd
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18216/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18216/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18216/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18216/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 13m 11s trunk passed +1 compile 1m 26s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 17s trunk passed +1 javadoc 1m 3s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 33s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 90m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 124m 8s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf38032a353b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5839fd Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18216/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18216/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18216/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18216/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 14m 26s trunk passed
          +1 compile 1m 38s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 43s the patch passed
          +1 javadoc 1m 2s the patch passed
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed.
          -1 unit 71m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          108m 21s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.TestBlockStoragePolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f049c6c24c8f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e023584
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18327/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18327/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18327/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18327/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 26s trunk passed +1 compile 1m 38s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 43s the patch passed +1 javadoc 1m 2s the patch passed +1 unit 1m 4s hadoop-hdfs-client in the patch passed. -1 unit 71m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 108m 21s Reason Tests Failed junit tests hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestBlockStoragePolicy Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f049c6c24c8f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e023584 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18327/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18327/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18327/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18327/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          +1, the patch looks good to me.

          Show
          jnp Jitendra Nath Pandey added a comment - +1, the patch looks good to me.
          Hide
          jingzhao Jing Zhao added a comment -

          Do you also want to take a look at the patch, Vinayakumar B?

          Show
          jingzhao Jing Zhao added a comment - Do you also want to take a look at the patch, Vinayakumar B ?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 12m 35s trunk passed
          +1 compile 1m 22s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 7s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 1m 18s the patch passed
          +1 javac 1m 18s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          +1 unit 63m 36s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          96m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 846b3da6b50e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e53f2b
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18378/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18378/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18378/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 12m 35s trunk passed +1 compile 1m 22s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 7s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 18s the patch passed +1 javac 1m 18s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project: The patch generated 1 new + 81 unchanged - 2 fixed = 82 total (was 83) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 18s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. +1 unit 63m 36s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 96m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848397/HDFS-8498.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 846b3da6b50e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e53f2b Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18378/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18378/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18378/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the review, Jitendra Nath Pandey! I will commit the patch shortly.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the review, Jitendra Nath Pandey ! I will commit the patch shortly.
          Hide
          jingzhao Jing Zhao added a comment -

          I've committed the patch into trunk.

          Show
          jingzhao Jing Zhao added a comment - I've committed the patch into trunk.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Jing Zhao very nice work!
          Do you plan to cherry pick the fix into 2.x branches?

          Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Jing Zhao very nice work! Do you plan to cherry pick the fix into 2.x branches? Thanks!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11258/)
          HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing (jing9: rev 627da6f7178e18aa41996969c408b6f344e297d1)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11258/ ) HDFS-8498 . Blocks can be committed with wrong size. Contributed by Jing (jing9: rev 627da6f7178e18aa41996969c408b6f344e297d1) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          jingzhao Jing Zhao added a comment -

          Wei-Chiu Chuang, currently I do not plan to backport this change to branch 2.x. But please feel free to do it if you think it's necessary and I will be happy to review.

          Show
          jingzhao Jing Zhao added a comment - Wei-Chiu Chuang , currently I do not plan to backport this change to branch 2.x. But please feel free to do it if you think it's necessary and I will be happy to review.
          Hide
          jingzhao Jing Zhao added a comment -

          Upload the patch for branch-2

          Show
          jingzhao Jing Zhao added a comment - Upload the patch for branch-2
          Hide
          jingzhao Jing Zhao added a comment -

          Reopen for the branch-2 patch.

          Show
          jingzhao Jing Zhao added a comment - Reopen for the branch-2 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 21m 45s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 7m 5s branch-2 passed
          +1 compile 1m 21s branch-2 passed with JDK v1.8.0_121
          +1 compile 1m 27s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 35s branch-2 passed
          +1 mvnsite 1m 28s branch-2 passed
          +1 mvneclipse 0m 31s branch-2 passed
          +1 findbugs 3m 39s branch-2 passed
          +1 javadoc 1m 16s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 2m 1s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 14s the patch passed with JDK v1.8.0_121
          +1 javac 1m 14s the patch passed
          +1 compile 1m 20s the patch passed with JDK v1.7.0_121
          +1 javac 1m 20s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 89 unchanged - 2 fixed = 91 total (was 91)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 56s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_121
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121.
          -1 unit 71m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          196m 6s



          Reason Tests
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853115/HDFS-8498.branch-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 351f1d8fde75 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / f07b52d
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18388/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18388/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18388/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18388/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 21m 45s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 7m 5s branch-2 passed +1 compile 1m 21s branch-2 passed with JDK v1.8.0_121 +1 compile 1m 27s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 35s branch-2 passed +1 mvnsite 1m 28s branch-2 passed +1 mvneclipse 0m 31s branch-2 passed +1 findbugs 3m 39s branch-2 passed +1 javadoc 1m 16s branch-2 passed with JDK v1.8.0_121 +1 javadoc 2m 1s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 14s the patch passed with JDK v1.8.0_121 +1 javac 1m 14s the patch passed +1 compile 1m 20s the patch passed with JDK v1.7.0_121 +1 javac 1m 20s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 89 unchanged - 2 fixed = 91 total (was 91) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_121 +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121. -1 unit 71m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 196m 6s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853115/HDFS-8498.branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 351f1d8fde75 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / f07b52d Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18388/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18388/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18388/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18388/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks very much for your branch-2 patch, Jing. I plan to review it today.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks very much for your branch-2 patch, Jing. I plan to review it today.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Jing Zhao thanks again for the branch-2 patch.
          The patch looks good, and the failed tests are not reproducible in my local tree.

          Can you fix the checkstyle issue?

          ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:1769: protected LocatedBlock locateFollowingBlock(DatanodeInfo[] excludedNodes,:62: 'excludedNodes' hides a field.

          +1 after the checkstyle issue is fixed.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Jing Zhao thanks again for the branch-2 patch. The patch looks good, and the failed tests are not reproducible in my local tree. Can you fix the checkstyle issue? ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:1769: protected LocatedBlock locateFollowingBlock(DatanodeInfo[] excludedNodes,:62: 'excludedNodes' hides a field. +1 after the checkstyle issue is fixed.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the review, Wei-Chiu Chuang. Update the branch-2 patch to address your comments.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the review, Wei-Chiu Chuang . Update the branch-2 patch to address your comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 8m 46s branch-2 passed
          +1 compile 1m 20s branch-2 passed with JDK v1.8.0_121
          +1 compile 1m 22s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 35s branch-2 passed
          +1 mvnsite 1m 26s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 3m 35s branch-2 passed
          +1 javadoc 1m 16s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 57s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 1m 12s the patch passed with JDK v1.8.0_121
          +1 javac 1m 12s the patch passed
          +1 compile 1m 20s the patch passed with JDK v1.7.0_121
          +1 javac 1m 20s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 1 new + 89 unchanged - 2 fixed = 90 total (was 91)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 58s the patch passed
          +1 javadoc 1m 12s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_121
          +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121.
          -1 unit 85m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          194m 2s



          Reason Tests
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
          JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854548/HDFS-8498.branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7261fcc1f555 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 29fe5af
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18446/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18446/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18446/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 46s branch-2 passed +1 compile 1m 20s branch-2 passed with JDK v1.8.0_121 +1 compile 1m 22s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 35s branch-2 passed +1 mvnsite 1m 26s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 3m 35s branch-2 passed +1 javadoc 1m 16s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 57s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 1m 12s the patch passed with JDK v1.8.0_121 +1 javac 1m 12s the patch passed +1 compile 1m 20s the patch passed with JDK v1.7.0_121 +1 javac 1m 20s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 1 new + 89 unchanged - 2 fixed = 90 total (was 91) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 58s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_121 +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121. -1 unit 85m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 194m 2s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854548/HDFS-8498.branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7261fcc1f555 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 29fe5af Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18446/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18446/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18446/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I pushed the commit to branch-2 and branch-2.8.
          There was a big refactory in Hadoop 2.7 so I didn't cherry picked the commit to branch-2.7.

          Thanks Jing Zhao!

          Show
          jojochuang Wei-Chiu Chuang added a comment - I pushed the commit to branch-2 and branch-2.8. There was a big refactory in Hadoop 2.7 so I didn't cherry picked the commit to branch-2.7. Thanks Jing Zhao !
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Sorry to reopen. Attach a branch-2.7 patch for precommit.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Sorry to reopen. Attach a branch-2.7 patch for precommit.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 35s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 41s branch-2.7 passed
          +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_121
          +1 compile 0m 59s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 25s branch-2.7 passed
          +1 mvnsite 0m 58s branch-2.7 passed
          +1 mvneclipse 0m 15s branch-2.7 passed
          +1 findbugs 2m 53s branch-2.7 passed
          +1 javadoc 0m 59s branch-2.7 passed with JDK v1.8.0_121
          +1 javadoc 1m 45s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.8.0_121
          +1 javac 0m 56s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_121
          +1 javac 0m 57s the patch passed
          -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 113 unchanged - 6 fixed = 117 total (was 119)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 3239 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 1m 21s The patch 77 line(s) with tabs.
          +1 findbugs 3m 4s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 43s the patch passed with JDK v1.7.0_121
          -1 unit 43m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          -1 asflicense 0m 19s The patch generated 3 ASF License warnings.
          133m 54s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855659/HDFS-8498.branch-2.7.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d01bd6c7be4a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / c8b1112
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18513/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18513/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 35s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s branch-2.7 passed +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_121 +1 compile 0m 59s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 0m 58s branch-2.7 passed +1 mvneclipse 0m 15s branch-2.7 passed +1 findbugs 2m 53s branch-2.7 passed +1 javadoc 0m 59s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 1m 45s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 52s the patch passed +1 compile 0m 56s the patch passed with JDK v1.8.0_121 +1 javac 0m 56s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_121 +1 javac 0m 57s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 113 unchanged - 6 fixed = 117 total (was 119) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 3239 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 21s The patch 77 line(s) with tabs. +1 findbugs 3m 4s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_121 -1 unit 43m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 19s The patch generated 3 ASF License warnings. 133m 54s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA JDK v1.7.0_121 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855659/HDFS-8498.branch-2.7.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d01bd6c7be4a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / c8b1112 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18513/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18513/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18513/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          There are a few genuine test failures in the branch-2.7 patch. I'll look into them.

          Show
          jojochuang Wei-Chiu Chuang added a comment - There are a few genuine test failures in the branch-2.7 patch. I'll look into them.
          Hide
          daryn Daryn Sharp added a comment -

          As the original filer, hardening the client wasn't the intended fix.

          The intention was to fix the NN ignoring the IBR-reported size and state when the block is on the expected storage. It only updates the GS.

               for (int i = 0; i < replicas.length; i++) {
                  DatanodeStorageInfo expected =
                      replicas[i].getExpectedStorageLocation();
                  if (expected == storage) {
                    replicas[i].setGenerationStamp(reportedBlock.getGenerationStamp());
                    return;
                  } else if (expected != null && expected.getDatanodeDescriptor() ==
          

          The NN wouldn't get into this bad state if it kept track of reported sizes and did some sanity checking.

          Show
          daryn Daryn Sharp added a comment - As the original filer, hardening the client wasn't the intended fix. The intention was to fix the NN ignoring the IBR-reported size and state when the block is on the expected storage. It only updates the GS. for ( int i = 0; i < replicas.length; i++) { DatanodeStorageInfo expected = replicas[i].getExpectedStorageLocation(); if (expected == storage) { replicas[i].setGenerationStamp(reportedBlock.getGenerationStamp()); return ; } else if (expected != null && expected.getDatanodeDescriptor() == The NN wouldn't get into this bad state if it kept track of reported sizes and did some sanity checking.
          Hide
          andrew.wang Andrew Wang added a comment -

          Is there a plan to resolve this JIRA? Been pending for about a month.

          Show
          andrew.wang Andrew Wang added a comment - Is there a plan to resolve this JIRA? Been pending for about a month.
          Hide
          zhz Zhe Zhang added a comment -

          I'd like to help get this committed to branch-2.7. Just triggered Jenkins again. Will look at UT failures if there are any.

          Show
          zhz Zhe Zhang added a comment - I'd like to help get this committed to branch-2.7. Just triggered Jenkins again. Will look at UT failures if there are any.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 17s branch-2.7 passed
          +1 compile 1m 4s branch-2.7 passed with JDK v1.8.0_131
          +1 compile 1m 5s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 26s branch-2.7 passed
          +1 mvnsite 1m 0s branch-2.7 passed
          +1 mvneclipse 0m 14s branch-2.7 passed
          -1 findbugs 2m 57s hadoop-hdfs-project/hadoop-hdfs in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 1m 3s branch-2.7 passed with JDK v1.8.0_131
          +1 javadoc 1m 49s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 56s the patch passed
          +1 compile 1m 1s the patch passed with JDK v1.8.0_131
          +1 javac 1m 1s the patch passed
          +1 compile 1m 3s the patch passed with JDK v1.7.0_121
          +1 javac 1m 3s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 114 unchanged - 6 fixed = 118 total (was 120)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 3675 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 1m 21s The patch 77 line(s) with tabs.
          +1 findbugs 3m 5s the patch passed
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_131
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_121
          -1 unit 56m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          -1 asflicense 0m 19s The patch generated 3 ASF License warnings.
          142m 50s



          Reason Tests
          JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-8498
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855659/HDFS-8498.branch-2.7.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f158ec12b672 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 8ad9efb
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19224/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19224/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 17s branch-2.7 passed +1 compile 1m 4s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 5s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 26s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 mvneclipse 0m 14s branch-2.7 passed -1 findbugs 2m 57s hadoop-hdfs-project/hadoop-hdfs in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 1m 3s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 49s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 56s the patch passed +1 compile 1m 1s the patch passed with JDK v1.8.0_131 +1 javac 1m 1s the patch passed +1 compile 1m 3s the patch passed with JDK v1.7.0_121 +1 javac 1m 3s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 114 unchanged - 6 fixed = 118 total (was 120) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 3675 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 21s The patch 77 line(s) with tabs. +1 findbugs 3m 5s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_121 -1 unit 56m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 19s The patch generated 3 ASF License warnings. 142m 50s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-8498 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855659/HDFS-8498.branch-2.7.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f158ec12b672 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 8ad9efb Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19224/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19224/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19224/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Zhe Zhang, would you consider resolving this jira and filing a separate one for the branch-2.7 commit?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Zhe Zhang , would you consider resolving this jira and filing a separate one for the branch-2.7 commit?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the suggestion Arpit Agarwal, I'm working on HDFS-11732 for the backport.

          Show
          zhz Zhe Zhang added a comment - Thanks for the suggestion Arpit Agarwal , I'm working on HDFS-11732 for the backport.
          Hide
          zhz Zhe Zhang added a comment -

          It's actually valuable to add a similar test as in HDFS-9289, but misreport the size instead of GS. Any taker?

          Show
          zhz Zhe Zhang added a comment - It's actually valuable to add a similar test as in HDFS-9289 , but misreport the size instead of GS. Any taker?
          Hide
          shv Konstantin Shvachko added a comment -

          Attaching latest backport to branch-2.7 patch from HDFS-11732.

          Show
          shv Konstantin Shvachko added a comment - Attaching latest backport to branch-2.7 patch from HDFS-11732 .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development