Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9220

Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Exception:
      2015-10-09 14:59:40 WARN DFSClient:1150 - fetchBlockByteRange(). Got a checksum exception for /tmp/file0.05355529331575182 at BP-353681639-10.10.10.10-1437493596883:blk_1075692769_9244882:0 from DatanodeInfoWithStorage[10.10.10.10]:5001

      All 3 replicas cause this exception and the read fails entirely with:
      BlockMissingException: Could not obtain block: BP-353681639-10.10.10.10-1437493596883:blk_1075692769_9244882 file=/tmp/file0.05355529331575182

      Code to reproduce is attached.
      Does not happen in 2.7.0.
      Data is read correctly if checksum verification is disabled.
      More generally, the failure happens when reading from the last block of a file and the last block has <= 512 bytes.

      1. test2.java
        1.0 kB
        Bogdan Raducanu
      2. HDFS-9220.002.patch
        3 kB
        Jing Zhao
      3. HDFS-9220.001.patch
        3 kB
        Jing Zhao
      4. HDFS-9220.000.patch
        3 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          Thanks Kihwal Lee! Agree that only minor conflict with HDFS-9220 for branch-2.7. In addition, 2.6 support java 6 also, so try-with-resource in test case has to be replaced. Just merge it to branch-2.6.

          Show
          djp Junping Du added a comment - Thanks Kihwal Lee ! Agree that only minor conflict with HDFS-9220 for branch-2.7. In addition, 2.6 support java 6 also, so try-with-resource in test case has to be replaced. Just merge it to branch-2.6.
          Hide
          kihwal Kihwal Lee added a comment -

          Junping Du I will upload patches for HDFS-4660 and HDFS-8722. HDFS-9220 seems to have only minor conflicts due to context.

          Show
          kihwal Kihwal Lee added a comment - Junping Du I will upload patches for HDFS-4660 and HDFS-8722 . HDFS-9220 seems to have only minor conflicts due to context.
          Hide
          djp Junping Du added a comment -

          Thanks Kihwal Lee for clarification on this. For layout fix, I think you mean HDFS-8791 (together with HDFS-8578). Isn't it? I am Ok with pulling that two fixes if it is possible to make on time.

          Show
          djp Junping Du added a comment - Thanks Kihwal Lee for clarification on this. For layout fix, I think you mean HDFS-8791 (together with HDFS-8578 ). Isn't it? I am Ok with pulling that two fixes if it is possible to make on time.
          Hide
          kihwal Kihwal Lee added a comment -

          This one goes with HDFS-4660 and HDFS-8722. HDFS-4660 fixed a data corruption issue and HDFS-8722 fixed a performance regression in HDFS-4660. These changes introduced this bug (HDFS-9220). So, if you want to fix the corruption issue, you need to pull in all three. The issue fixed in HDFS-4660 was originally seen very rarely, but after the datanode layout change, it was occurring much more frequently along with other bugs we fixed regarding pipeline failures. If you plan to pull in the layout fix in the next release of 2.6, HDFS-4660 may not be as critical.

          Show
          kihwal Kihwal Lee added a comment - This one goes with HDFS-4660 and HDFS-8722 . HDFS-4660 fixed a data corruption issue and HDFS-8722 fixed a performance regression in HDFS-4660 . These changes introduced this bug ( HDFS-9220 ). So, if you want to fix the corruption issue, you need to pull in all three. The issue fixed in HDFS-4660 was originally seen very rarely, but after the datanode layout change, it was occurring much more frequently along with other bugs we fixed regarding pipeline failures. If you plan to pull in the layout fix in the next release of 2.6, HDFS-4660 may not be as critical.
          Hide
          djp Junping Du added a comment -

          Hi Jing Zhao and Kihwal Lee, shall we cherry-pick the fix to branch-2.6 as well?

          Show
          djp Junping Du added a comment - Hi Jing Zhao and Kihwal Lee , shall we cherry-pick the fix to branch-2.6 as well?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2440 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2440/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2440 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2440/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/502/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/502/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #538 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/538/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #538 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/538/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1274/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1274/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2487 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2487/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2487 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2487/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #551 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/551/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #551 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/551/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8645/)
          HDFS-9220. Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8645/ ) HDFS-9220 . Reading small file (< 512 bytes) that is open for append (kihwal: rev c7c36cbd6218f46c33d7fb2f60cd52cb29e6d720) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk, branch-2 and branch-2.7. Thanks for reporting the bug, Bogdan Raducanu. Thanks for quickly analyzing and fixing it, Jing Zhao.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.7. Thanks for reporting the bug, Bogdan Raducanu . Thanks for quickly analyzing and fixing it, Jing Zhao .
          Hide
          kihwal Kihwal Lee added a comment -

          Test failures seem unrelated. They failed due to java.lang.NoClassDefFoundError, Port in Use, etc., which are test runtime issue.
          +1 the patch looks good to me.

          Show
          kihwal Kihwal Lee added a comment - Test failures seem unrelated. They failed due to java.lang.NoClassDefFoundError , Port in Use , etc., which are test runtime issue. +1 the patch looks good to me.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 9s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 57s There were no new javac warning messages.
          +1 javadoc 11m 26s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 60, now 60).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 53m 52s Tests failed in hadoop-hdfs.
              106m 11s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.web.TestWebHDFSXAttr
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.web.resources.TestParam
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          Timed out tests org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
            org.apache.hadoop.hdfs.web.TestWebHDFSForHA



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766603/HDFS-9220.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3d50855
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12989/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12989/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 9s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 57s There were no new javac warning messages. +1 javadoc 11m 26s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 60, now 60). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 53m 52s Tests failed in hadoop-hdfs.     106m 11s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.web.resources.TestParam   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.fs.TestGlobPaths   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Timed out tests org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage   org.apache.hadoop.hdfs.web.TestWebHDFSForHA Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766603/HDFS-9220.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3d50855 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12989/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12989/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12989/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 24m 57s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 9m 16s There were no new javac warning messages.
          +1 javadoc 11m 50s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 52s The applied patch generated 1 new checkstyle issues (total was 60, now 60).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 40s mvn install still works.
          -1 eclipse:eclipse 0m 35s The patch failed to build with eclipse:eclipse.
          +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 32s Pre-build of native portion
          -1 hdfs tests 53m 15s Tests failed in hadoop-hdfs.
              110m 17s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.fs.TestGlobPaths



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766603/HDFS-9220.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 56dc777
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12984/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12984/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 24m 57s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 16s There were no new javac warning messages. +1 javadoc 11m 50s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 52s The applied patch generated 1 new checkstyle issues (total was 60, now 60). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. -1 eclipse:eclipse 0m 35s The patch failed to build with eclipse:eclipse. +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 32s Pre-build of native portion -1 hdfs tests 53m 15s Tests failed in hadoop-hdfs.     110m 17s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.fs.TestGlobPaths Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766603/HDFS-9220.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 56dc777 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12984/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12984/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12984/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Looks like most of the test failures were caused by NULL checksum type. When the checksum is disabled, offset and end are both 0 but the assertion crcBytes != 0 fails. Update the patch to fix this part.

          Show
          jingzhao Jing Zhao added a comment - Looks like most of the test failures were caused by NULL checksum type. When the checksum is disabled, offset and end are both 0 but the assertion crcBytes != 0 fails. Update the patch to fix this part.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 57s There were no new javac warning messages.
          +1 javadoc 10m 25s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 27s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 62m 53s Tests failed in hadoop-hdfs.
              109m 17s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestSetrepIncreasing
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestBlockReaderLocal
          Timed out tests org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
            org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.server.namenode.TestAuditLogs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766442/HDFS-9220.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d6c8bad
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12978/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12978/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12978/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12978/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 10m 25s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 27s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 62m 53s Tests failed in hadoop-hdfs.     109m 17s   Reason Tests Failed unit tests hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestBlockReaderLocal Timed out tests org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.server.namenode.TestAuditLogs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766442/HDFS-9220.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d6c8bad Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12978/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12978/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12978/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12978/console This message was automatically generated.
          Hide
          bograd Bogdan Raducanu added a comment -

          Jing Zhao,

          Reading the last block which is <= 512 bytes is a problem (so it affects bigger files too). Can you think of other cases where it will fail? I'm trying to find a workaround that doesn't require completely disabling checksum verification.

          Thank you.

          Show
          bograd Bogdan Raducanu added a comment - Jing Zhao , Reading the last block which is <= 512 bytes is a problem (so it affects bigger files too). Can you think of other cases where it will fail? I'm trying to find a workaround that doesn't require completely disabling checksum verification. Thank you.
          Hide
          jingzhao Jing Zhao added a comment -

          The failed tests look like related. Will dig further.

          Show
          jingzhao Jing Zhao added a comment - The failed tests look like related. Will dig further.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 24m 15s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 10m 26s There were no new javac warning messages.
          +1 javadoc 13m 38s There were no new javadoc warning messages.
          +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 58s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 58s mvn install still works.
          +1 eclipse:eclipse 0m 48s The patch built with eclipse:eclipse.
          +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 5m 6s Pre-build of native portion
          -1 hdfs tests 86m 25s Tests failed in hadoop-hdfs.
              148m 26s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestSetrepIncreasing
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestBPOfferService
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
          Timed out tests org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766442/HDFS-9220.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 40cac59
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12968/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12968/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12968/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12968/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 24m 15s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 26s There were no new javac warning messages. +1 javadoc 13m 38s There were no new javadoc warning messages. +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 58s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 58s mvn install still works. +1 eclipse:eclipse 0m 48s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 5m 6s Pre-build of native portion -1 hdfs tests 86m 25s Tests failed in hadoop-hdfs.     148m 26s   Reason Tests Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestBPOfferService   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog Timed out tests org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766442/HDFS-9220.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 40cac59 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12968/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12968/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12968/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12968/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 23s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 56s There were no new javac warning messages.
          +1 javadoc 11m 30s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 36s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 2m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 40s Pre-build of native portion
          -1 hdfs tests 66m 6s Tests failed in hadoop-hdfs.
              117m 40s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.TestSnapshotCommands
            hadoop.hdfs.TestFileCreationClient
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.TestDFSPermission
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.TestSetrepIncreasing
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestBlockReaderLocal
          Timed out tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead
            org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.TestDatanodeReport



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766428/HDFS-9220.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 40cac59
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12965/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12965/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12965/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12965/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 23s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 56s There were no new javac warning messages. +1 javadoc 11m 30s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 36s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 2m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 40s Pre-build of native portion -1 hdfs tests 66m 6s Tests failed in hadoop-hdfs.     117m 40s   Reason Tests Failed unit tests hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.TestFileCreationClient   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestBlockReaderLocal Timed out tests org.apache.hadoop.hdfs.TestParallelShortCircuitRead   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.TestDatanodeReport Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766428/HDFS-9220.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 40cac59 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12965/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12965/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12965/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12965/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Update the patch to address Daniel Templeton's comments.

          Show
          jingzhao Jing Zhao added a comment - Update the patch to address Daniel Templeton 's comments.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, Jing Zhao. Could you add some comments to the patch to make it a little more clear what the test is doing? Something like:

          // Reading in data from the small block. If the checksum is wrong, the read will throw an exception.  If it doesn't, the checksum is correct.
          

          I'd also love a blank line before the try to make it easier on the eyes.

          Show
          templedf Daniel Templeton added a comment - Thanks, Jing Zhao . Could you add some comments to the patch to make it a little more clear what the test is doing? Something like: // Reading in data from the small block. If the checksum is wrong, the read will throw an exception. If it doesn't, the checksum is correct. I'd also love a blank line before the try to make it easier on the eyes.
          Hide
          jingzhao Jing Zhao added a comment -

          Submit a patch including the above fix and also Bogdan Raducanu's unit test.

          Show
          jingzhao Jing Zhao added a comment - Submit a patch including the above fix and also Bogdan Raducanu 's unit test.
          Hide
          kihwal Kihwal Lee added a comment -

          Looks like the following change can fix the issue:

          Yes, that fixes the issue. Since it is a critical issue and the fix is already known, I think Jing Zhao can take over. Jagadesh Kiran N, if you are not going to provide a patch soon, please let Jing take over this jira.

          Show
          kihwal Kihwal Lee added a comment - Looks like the following change can fix the issue: Yes, that fixes the issue. Since it is a critical issue and the fix is already known, I think Jing Zhao can take over. Jagadesh Kiran N , if you are not going to provide a patch soon, please let Jing take over this jira.
          Hide
          templedf Daniel Templeton added a comment -

          My testing shows that change fixes the issue I see.

          Show
          templedf Daniel Templeton added a comment - My testing shows that change fixes the issue I see.
          Hide
          jingzhao Jing Zhao added a comment -

          Looks like the following change can fix the issue:

          --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          @@ -756,7 +756,7 @@ private int receivePacket() throws IOException {
                       // If offset > end, there is no more checksum to write.
                       // I.e. a partial chunk checksum rewrite happened and there is no
                       // more to write after that.
          -            if (offset > end) {
          +            if (offset >= end) {
                         assert crcBytes != null;
                         lastCrc = crcBytes;
                       } else {
          

          As the javadoc says, when offset==end, there is no more checksum to write thus the lastCrc should be set to crcBytes.

          Show
          jingzhao Jing Zhao added a comment - Looks like the following change can fix the issue: --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java @@ -756,7 +756,7 @@ private int receivePacket() throws IOException { // If offset > end, there is no more checksum to write. // I.e. a partial chunk checksum rewrite happened and there is no // more to write after that. - if (offset > end) { + if (offset >= end) { assert crcBytes != null ; lastCrc = crcBytes; } else { As the javadoc says, when offset==end, there is no more checksum to write thus the lastCrc should be set to crcBytes.
          Hide
          templedf Daniel Templeton added a comment -

          I saw a similar issue, and it went away when I backed out HDFS-4660 and HDFS-8722. Might be the same issue.

          Show
          templedf Daniel Templeton added a comment - I saw a similar issue, and it went away when I backed out HDFS-4660 and HDFS-8722 . Might be the same issue.
          Hide
          kihwal Kihwal Lee added a comment -

          It looks like it fails only when append is involved. If it is closed after appending, read succeeds.

          action after writing 200 bytes writing 300 bytes read result
          close append & hflush fail
          close append & close success
          hflush close success
          hflush hflush success
          Show
          kihwal Kihwal Lee added a comment - It looks like it fails only when append is involved. If it is closed after appending, read succeeds. action after writing 200 bytes writing 300 bytes read result close append & hflush fail close append & close success hflush close success hflush hflush success
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Bogdan Raducanu thanks for reporting the issue, I will try to analyze this, if you have already planned a patch you can allocate it to you

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Bogdan Raducanu thanks for reporting the issue, I will try to analyze this, if you have already planned a patch you can allocate it to you
          Hide
          bograd Bogdan Raducanu added a comment -

          perhaps related to this

          Show
          bograd Bogdan Raducanu added a comment - perhaps related to this

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              bograd Bogdan Raducanu
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development