HBase
  1. HBase
  2. HBASE-5074

support checksums in HBase block cache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.checksum.algorithm, name of an algorithm that is used to compute checksums.

      You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
      Show
      Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.checksum.algorithm, name of an algorithm that is used to compute checksums. You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
    • Tags:
      0.96notable

      Description

      The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers.

      1. 5074-0.94.txt
        214 kB
        Lars Hofhansl
      2. ASF.LICENSE.NOT.GRANTED--D1521.14.patch
        213 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D1521.14.patch
        213 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D1521.13.patch
        213 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D1521.13.patch
        213 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D1521.12.patch
        213 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--D1521.12.patch
        213 kB
        Phabricator
      8. ASF.LICENSE.NOT.GRANTED--D1521.11.patch
        213 kB
        Phabricator
      9. ASF.LICENSE.NOT.GRANTED--D1521.11.patch
        213 kB
        Phabricator
      10. D1521.10.patch
        210 kB
        stack
      11. D1521.10.patch
        210 kB
        stack
      12. D1521.10.patch
        210 kB
        stack
      13. ASF.LICENSE.NOT.GRANTED--D1521.10.patch
        210 kB
        Phabricator
      14. ASF.LICENSE.NOT.GRANTED--D1521.10.patch
        210 kB
        Phabricator
      15. ASF.LICENSE.NOT.GRANTED--D1521.9.patch
        210 kB
        Phabricator
      16. ASF.LICENSE.NOT.GRANTED--D1521.9.patch
        210 kB
        Phabricator
      17. ASF.LICENSE.NOT.GRANTED--D1521.8.patch
        209 kB
        Phabricator
      18. ASF.LICENSE.NOT.GRANTED--D1521.8.patch
        209 kB
        Phabricator
      19. ASF.LICENSE.NOT.GRANTED--D1521.7.patch
        209 kB
        Phabricator
      20. ASF.LICENSE.NOT.GRANTED--D1521.7.patch
        209 kB
        Phabricator
      21. ASF.LICENSE.NOT.GRANTED--D1521.6.patch
        209 kB
        Phabricator
      22. ASF.LICENSE.NOT.GRANTED--D1521.6.patch
        209 kB
        Phabricator
      23. ASF.LICENSE.NOT.GRANTED--D1521.5.patch
        205 kB
        Phabricator
      24. ASF.LICENSE.NOT.GRANTED--D1521.5.patch
        205 kB
        Phabricator
      25. ASF.LICENSE.NOT.GRANTED--D1521.4.patch
        204 kB
        Phabricator
      26. ASF.LICENSE.NOT.GRANTED--D1521.4.patch
        204 kB
        Phabricator
      27. ASF.LICENSE.NOT.GRANTED--D1521.3.patch
        218 kB
        Phabricator
      28. ASF.LICENSE.NOT.GRANTED--D1521.3.patch
        218 kB
        Phabricator
      29. ASF.LICENSE.NOT.GRANTED--D1521.2.patch
        188 kB
        Phabricator
      30. ASF.LICENSE.NOT.GRANTED--D1521.2.patch
        188 kB
        Phabricator
      31. ASF.LICENSE.NOT.GRANTED--D1521.1.patch
        155 kB
        Phabricator
      32. ASF.LICENSE.NOT.GRANTED--D1521.1.patch
        155 kB
        Phabricator

        Issue Links

          Activity

          stack made changes -
          Tags 0.96notable
          Anoop Sam John made changes -
          Release Note Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.bytes.per.checksum, name of an algorithm that is used to compute checksums.

          You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
          Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.checksum.algorithm, name of an algorithm that is used to compute checksums.

          You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Lars Hofhansl made changes -
          Link This issue relates to HBASE-6868 [ HBASE-6868 ]
          stack made changes -
          Release Note Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.bytes.per.checksum, name of an algorithm that is used to compute checksums.

          You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
          Lars Hofhansl made changes -
          Link This issue relates to HBASE-5864 [ HBASE-5864 ]
          Lars Hofhansl made changes -
          Link This issue relates to HBASE-5720 [ HBASE-5720 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Lars Hofhansl made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lars Hofhansl made changes -
          Attachment 5074-0.94.txt [ 12517648 ]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Phabricator made changes -
          Attachment D1521.14.patch [ 12517351 ]
          Phabricator made changes -
          Attachment D1521.14.patch [ 12517350 ]
          Phabricator made changes -
          Attachment D1521.13.patch [ 12517204 ]
          Phabricator made changes -
          Attachment D1521.13.patch [ 12517203 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          Ted Yu made changes -
          Comment [ dhruba updated the revision "[jira] [HBASE-5074] Support checksums in HBase block cache".
          Reviewers: mbautin

            Fixed failed unit test TestFixedFileTrailer

          REVISION DETAIL
            https://reviews.facebook.net/D1521

          AFFECTED FILES
            src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
            src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java
            src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
            src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
            src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
            src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
            src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
            src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
            src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
            src/main/java/org/apache/hadoop/hbase/HConstants.java
            src/main/java/org/apache/hadoop/hbase/fs
            src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
            src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
            src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
            src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java
            src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
            src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
            src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
            src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
            src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
            src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
            src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
          ]
          Phabricator made changes -
          Attachment D1521.12.patch [ 12516807 ]
          Phabricator made changes -
          Attachment D1521.12.patch [ 12516806 ]
          Phabricator made changes -
          Attachment D1521.11.patch [ 12516798 ]
          Phabricator made changes -
          Attachment D1521.11.patch [ 12516797 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          stack made changes -
          Attachment D1521.10.patch [ 12516208 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          stack made changes -
          Attachment D1521.10.patch [ 12516189 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          stack made changes -
          Attachment D1521.10.patch [ 12516181 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Phabricator made changes -
          Attachment D1521.10.patch [ 12516146 ]
          Phabricator made changes -
          Attachment D1521.10.patch [ 12516145 ]
          Phabricator made changes -
          Attachment D1521.9.patch [ 12515829 ]
          Phabricator made changes -
          Attachment D1521.9.patch [ 12515828 ]
          Phabricator made changes -
          Attachment D1521.8.patch [ 12515642 ]
          Phabricator made changes -
          Attachment D1521.8.patch [ 12515641 ]
          Phabricator made changes -
          Attachment D1521.7.patch [ 12515551 ]
          Phabricator made changes -
          Attachment D1521.7.patch [ 12515550 ]
          Phabricator made changes -
          Attachment D1521.6.patch [ 12514759 ]
          Phabricator made changes -
          Attachment D1521.6.patch [ 12514758 ]
          Phabricator made changes -
          Attachment D1521.5.patch [ 12513780 ]
          Phabricator made changes -
          Attachment D1521.5.patch [ 12513779 ]
          Phabricator made changes -
          Attachment D1521.4.patch [ 12513715 ]
          Phabricator made changes -
          Attachment D1521.4.patch [ 12513714 ]
          Ted Yu made changes -
          Comment [ tedyu has commented on the revision "[jira] [HBASE-5074] Support checksums in HBase block cache".

          INLINE COMMENTS
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java:425 This cast is not safe. See https://builds.apache.org/job/PreCommit-HBASE-Build/907//testReport/org.apache.hadoop.hbase.mapreduce/TestLoadIncrementalHFiles/testSimpleLoad/:

            Caused by: java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to org.apache.hadoop.hbase.util.HFileSystem
             at org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:425)
             at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:433)
             at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:407)
             at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:328)
             at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:326)
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java:160 Should we default to CRC32C ?
            src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java:2 No year is needed.
            src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java:59 Shall we name this variable ctor ?

            Similar comment applies to other meth variables in this patch.

          REVISION DETAIL
            https://reviews.facebook.net/D1521
          ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Phabricator made changes -
          Attachment D1521.3.patch [ 12513416 ]
          Phabricator made changes -
          Attachment D1521.3.patch [ 12513415 ]
          Phabricator made changes -
          Attachment D1521.2.patch [ 12513016 ]
          Phabricator made changes -
          Attachment D1521.2.patch [ 12513015 ]
          Ted Yu made changes -
          Comment [ mbautin has commented on the revision "[jira] [HBASE-5074] Support checksums in HBase block cache".
          Added CCs: dhruba

            @Dhruba: The "checksum at the end of block" approach seems reasonable and the implementation looks good! Specific comments inline.

          INLINE COMMENTS
            src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java:357 What is the purpose of the hfs parameter here?
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:49 s/preceeding/preceding/
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:50 s/deermines/determines/
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:51 s/does not need/do not need/
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:119 s/major/minor/
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:260 Rename the existing expectVersion to expectMajorVersion for clarity.
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:343 Rename to expectMajorVersion for clarity.

            Also, does the version field of this class now only contain the major version? If so, rename it to majorVersion.
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:345 Add the word "major" to the error message.
            src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java:462 Rename to getMajorVersion
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java:402 Can we modify the parameter type and get rid of this cast?
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java:415 This is not a constructor, but a factory method.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java:417 Add "ForTest" to method name for clarity.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:91 s/has/have/
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:95 Consider replacing the "_V0" suffix with something more meaningful like "_NO_CHECKSUM".
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:102 Consider using a suffix "_WITH_CHECKSUM".
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:123 When the number of bytes per checksum becomes configurable, will that require a persistent data format change? What will the upgrade procedure be in that case?
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:138 It is not clear from this call that 0 is minor version. Create a constant with a meaningful name (e.g. MINOR_VERSION_NO_CHECKSUM).
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:149 Consider adding "WithChecksum" to variable name.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:395-398 This is becoming bulky. Factor out the common term (uncompressedSizeWithoutHeader + headerSize() + cksumBytes) into a local variable. Also avoid evaluating headerSize() twice.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:400-402 Reuse the new local variables from the above comments here.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:441-442 Update this comment, since the meaning of "extraBytes" has changed from just being the room for the next block's header to a much more complex role.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:757-758 Should we throw an IOException instead since this method already throws it?
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:771-772 tmp is a particularly bad variable name. Combine these two lines and get rid of tmp.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:808-809 Get rid of tmp and combine these two lines.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:788 This method and the above one seem to share a lot of code. Is it possible to get rid of code duplication?

            Also, these two methods seem isolated enough to be moved to another class, maybe even as static methods.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:866-870 Do we need this in case of minorVersion = 0? Or do we always write new files with checksums?
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:973-975 Somehow the fact that checksum format is different for compressed and uncompressed blocks has escaped me halfway through the review. Maybe it is worth explicitly mentioning that in javadoc.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:999-1011 Use System.arraycopy instead of loops.

            Add "ForTest" to method name to discourage its use in production.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1202 Nice! Thanks for locking down these internal base classes and methods.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1389-1390 Delete one of these lines.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1520-1527 Does it make sense to move this checksum instantiation code to a function and reuse it everywhere we call ChecksumFactory.newInstance()?
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1786 Remove this and other debug output statements.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1871 As I mentioned, it is probably better to move checksum computation and validation code to a separate utility class.
            src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java:220 Use a constant to indicate that this is a minor version without checksum support instead of just 0.
            src/main/java/org/apache/hadoop/hbase/util/HFileSystem.java:56 Is this necessary? Does not Java call default constructors automatically?
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:70 This is for testing only, right?
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java:225 Long line.
            src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java:1 A lot in this file appears to be copy-paste from TestHFileBlock, so it very difficult to see the real changes. Please reuse the appropriate code instead.

          REVISION DETAIL
            https://reviews.facebook.net/D1521
          ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12512350/D1521.1.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 50 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.regionserver.TestStore
                            org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
                            org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
                            org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit
                            org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
                            org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
                            org.apache.hadoop.hbase.io.hfile.TestHFileBlock
                            org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
                            org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
                            org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
                            org.apache.hadoop.hbase.util.TestMergeTool
                            org.apache.hadoop.hbase.util.TestMergeTable
                            org.apache.hadoop.hbase.mapreduce.TestImportTsv
                            org.apache.hadoop.hbase.mapred.TestTableMapReduce
                            org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
                            org.apache.hadoop.hbase.coprocessor.TestWALObserver

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/867//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/867//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/867//console

          This message is automatically generated. ]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Comment [ tedyu has commented on the revision "[jira] [HBASE-5074] Support checksums in HBase block cache".

            Good job, Dhruba.

            I like this comment from HFileSystem:

            + * In future, if we want to make hlogs be in a different filesystem,
            + * this is the place to make it happen.

            I only see one setVerifyChecksum() call in the HFileSystem ctor.
            The readfs is used by createReaderWithEncoding().

            Shall we give readfs more flexibility where checksum verification can be configured dynamically ?

          REVISION DETAIL
            https://reviews.facebook.net/D1521
          ]
          Phabricator made changes -
          Attachment D1521.1.patch [ 12512350 ]
          Phabricator made changes -
          Attachment D1521.1.patch [ 12512349 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-2699 [ HDFS-2699 ]
          dhruba borthakur created issue -

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development