Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8722

Optimize datanode writes for small writes and flushes

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After the data corruption fix by HDFS-4660, the CRC recalculation for partial chunk is executed more frequently, if the client repeats writing few bytes and calling hflush/hsync. This is because the generic logic forces CRC recalculation if on-disk data is not CRC chunk aligned. Prior to HDFS-4660, datanode blindly accepted whatever CRC client provided, if the incoming data is chunk-aligned. This was the source of the corruption.

      We can still optimize for the most common case where a client is repeatedly writing small number of bytes followed by hflush/hsync with no pipeline recovery or append, by allowing the previous behavior for this specific case. If the incoming data has a duplicate portion and that is at the last chunk-boundary before the partial chunk on disk, datanode can use the checksum supplied by the client without redoing the checksum on its own. This reduces disk reads as well as CPU load for the checksum calculation.

      If the incoming packet data goes back further than the last on-disk chunk boundary, datanode will still do a recalculation, but this occurs rarely during pipeline recoveries. Thus the optimization for this specific case should be sufficient to speed up the vast majority of cases.

      1. HDFS-8722.patch
        4 kB
        Kihwal Lee
      2. HDFS-8722.v1.patch
        4 kB
        Kihwal Lee
      3. HDFS-8722.br26.patch
        5 kB
        Kihwal Lee

        Activity

        Hide
        djp Junping Du added a comment -

        I have merge the 2.6 patch to branch-2.6. Thanks Kihwal Lee for help on this!

        Show
        djp Junping Du added a comment - I have merge the 2.6 patch to branch-2.6. Thanks Kihwal Lee for help on this!
        Hide
        kihwal Kihwal Lee added a comment -

        Attaching 2.6 version of the patch.

        Show
        kihwal Kihwal Lee added a comment - Attaching 2.6 version of the patch.
        Hide
        djp Junping Du added a comment -

        Hi Kihwal Lee, shall we backport the fix to 2.6 branch also?

        Show
        djp Junping Du added a comment - Hi Kihwal Lee , shall we backport the fix to 2.6 branch also?
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2203/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2203/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/255/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/255/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/245/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/245/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2184/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2184/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #987 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/987/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #987 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/987/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #257 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/257/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #257 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/257/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8163/)
        HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8163/ ) HDFS-8722 . Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee (kihwal: rev 59388a801514d6af64ef27fbf246d8054f1dcc74) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        kihwal Kihwal Lee added a comment -

        Thanks for the review, Arpit. I've committed this to trunk, branch-2 and branch-2.7.

        Show
        kihwal Kihwal Lee added a comment - Thanks for the review, Arpit. I've committed this to trunk, branch-2 and branch-2.7.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 for the patch.

        Verified that it brings unaligned writes on par with 2.6.0.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 for the patch. Verified that it brings unaligned writes on par with 2.6.0.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Nice catch. A quick test shows ~15% throughput regression since 2.6.0 for [small writes+hflush].

        Release IO Size Throughput (KBps)
        2.6.0 16411 22263
        2.7.1 16411 18609

        Chunk aligned writes are at par with 2.6.0.

        Release IO Size Throughput (KBps)
        2.6.0 16384 21651
        2.7.1 16384 21460

        1GB file with 128 MB blocks.

        Show
        arpitagarwal Arpit Agarwal added a comment - Nice catch. A quick test shows ~15% throughput regression since 2.6.0 for [small writes+hflush]. Release IO Size Throughput (KBps) 2.6.0 16411 22263 2.7.1 16411 18609 Chunk aligned writes are at par with 2.6.0. Release IO Size Throughput (KBps) 2.6.0 16384 21651 2.7.1 16384 21460 1GB file with 128 MB blocks.
        Hide
        kihwal Kihwal Lee added a comment -

        The latest patch restores the small write/hflush performance comparable to the pre-HDFS-4660 level, less the corruption bug.

        Checkstyle:

        BlockReceiver.java:470:3: Method length is 300 lines (max allowed is 150)
        

        The method was already longer than the "limit". Excluding comments, I removed 5 lines and added 7 lines.

        No new test case is needed. Existing test cases do fully exercise the code path by performing small writes/flush or small appends.

        Show
        kihwal Kihwal Lee added a comment - The latest patch restores the small write/hflush performance comparable to the pre- HDFS-4660 level, less the corruption bug. Checkstyle: BlockReceiver.java:470:3: Method length is 300 lines (max allowed is 150) The method was already longer than the "limit". Excluding comments, I removed 5 lines and added 7 lines. No new test case is needed. Existing test cases do fully exercise the code path by performing small writes/flush or small appends.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 1s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 20s The applied patch generated 1 new checkstyle issues (total was 60, now 60).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 19s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 3s Pre-build of native portion
        +1 hdfs tests 157m 32s Tests passed in hadoop-hdfs.
            200m 56s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744350/HDFS-8722.v1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 2e3d83f
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11632/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11632/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 1s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 1 new checkstyle issues (total was 60, now 60). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 3s Pre-build of native portion +1 hdfs tests 157m 32s Tests passed in hadoop-hdfs.     200m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744350/HDFS-8722.v1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2e3d83f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11632/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11632/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11632/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        Forgot to remove one line in the patch.

        Show
        kihwal Kihwal Lee added a comment - Forgot to remove one line in the patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 47s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 17s The applied patch generated 1 new checkstyle issues (total was 60, now 60).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 21s Pre-build of native portion
        -1 hdfs tests 162m 34s Tests failed in hadoop-hdfs.
            210m 3s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestFileAppend4
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          hadoop.hdfs.TestAppendSnapshotTruncate
          hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.TestFileAppend2
          hadoop.hdfs.TestGetFileChecksum
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.TestDataTransferProtocol
          hadoop.hdfs.web.TestWebHdfsFileSystemContract
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          hadoop.hdfs.TestAppendDifferentChecksum
          hadoop.hdfs.TestHDFSFileSystemContract
          hadoop.hdfs.TestFileAppend3
          hadoop.hdfs.server.namenode.ha.TestHAAppend



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744005/HDFS-8722.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7e2fe8c
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11605/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11605/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 17s The applied patch generated 1 new checkstyle issues (total was 60, now 60). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 21s Pre-build of native portion -1 hdfs tests 162m 34s Tests failed in hadoop-hdfs.     210m 3s   Reason Tests Failed unit tests hadoop.hdfs.TestFileAppend4   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744005/HDFS-8722.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7e2fe8c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11605/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11605/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11605/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        The patch distinguishes the conditions for recalculating CRC from overwriting the last checksum. If the on-disk data is not chunk-aligned, the checksum is overwritten, but checksum recalculation will be done only when the incoming data does not start at the last chunk boundary. Otherwise, the client supplied chunk checksum will be used for overwriting the last partial chunk checksum.

        Show
        kihwal Kihwal Lee added a comment - The patch distinguishes the conditions for recalculating CRC from overwriting the last checksum. If the on-disk data is not chunk-aligned, the checksum is overwritten, but checksum recalculation will be done only when the incoming data does not start at the last chunk boundary. Otherwise, the client supplied chunk checksum will be used for overwriting the last partial chunk checksum.

          People

          • Assignee:
            kihwal Kihwal Lee
            Reporter:
            kihwal Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development