Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5074

support checksums in HBase block cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.94.0, 0.95.0
    • regionserver
    • None
    • Reviewed
    • Hide
      Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.checksum.algorithm, name of an algorithm that is used to compute checksums.

      You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
      Show
      Adds hbase.regionserver.checksum.verify. If hbase.regionserver.checksum.verify is set to true, then hbase will read data and then verify checksums. Checksum verification inside hdfs will be switched off. If the hbase-checksum verification fails, then it will switch back to using hdfs checksums for verifiying data that is being read from storage. Also adds hbase.hstore.bytes.per.checksum -- number of bytes in a newly created checksum chunk -- and hbase.hstore.checksum.algorithm, name of an algorithm that is used to compute checksums. You will currently only see benefit if you have the local read short-circuit enabled -- see http://hbase.apache.org/book.html#perf.hdfs.configs -- while HDFS-3429 goes unfixed.
    • 0.96notable

    Description

      The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers.

      Attachments

        1. 5074-0.94.txt
          214 kB
          Lars Hofhansl
        2. ASF.LICENSE.NOT.GRANTED--D1521.1.patch
          155 kB
          Phabricator
        3. ASF.LICENSE.NOT.GRANTED--D1521.1.patch
          155 kB
          Phabricator
        4. ASF.LICENSE.NOT.GRANTED--D1521.10.patch
          210 kB
          Phabricator
        5. ASF.LICENSE.NOT.GRANTED--D1521.10.patch
          210 kB
          Phabricator
        6. ASF.LICENSE.NOT.GRANTED--D1521.11.patch
          213 kB
          Phabricator
        7. ASF.LICENSE.NOT.GRANTED--D1521.11.patch
          213 kB
          Phabricator
        8. ASF.LICENSE.NOT.GRANTED--D1521.12.patch
          213 kB
          Phabricator
        9. ASF.LICENSE.NOT.GRANTED--D1521.12.patch
          213 kB
          Phabricator
        10. ASF.LICENSE.NOT.GRANTED--D1521.13.patch
          213 kB
          Phabricator
        11. ASF.LICENSE.NOT.GRANTED--D1521.13.patch
          213 kB
          Phabricator
        12. ASF.LICENSE.NOT.GRANTED--D1521.14.patch
          213 kB
          Phabricator
        13. ASF.LICENSE.NOT.GRANTED--D1521.14.patch
          213 kB
          Phabricator
        14. ASF.LICENSE.NOT.GRANTED--D1521.2.patch
          188 kB
          Phabricator
        15. ASF.LICENSE.NOT.GRANTED--D1521.2.patch
          188 kB
          Phabricator
        16. ASF.LICENSE.NOT.GRANTED--D1521.3.patch
          218 kB
          Phabricator
        17. ASF.LICENSE.NOT.GRANTED--D1521.3.patch
          218 kB
          Phabricator
        18. ASF.LICENSE.NOT.GRANTED--D1521.4.patch
          204 kB
          Phabricator
        19. ASF.LICENSE.NOT.GRANTED--D1521.4.patch
          204 kB
          Phabricator
        20. ASF.LICENSE.NOT.GRANTED--D1521.5.patch
          205 kB
          Phabricator
        21. ASF.LICENSE.NOT.GRANTED--D1521.5.patch
          205 kB
          Phabricator
        22. ASF.LICENSE.NOT.GRANTED--D1521.6.patch
          209 kB
          Phabricator
        23. ASF.LICENSE.NOT.GRANTED--D1521.6.patch
          209 kB
          Phabricator
        24. ASF.LICENSE.NOT.GRANTED--D1521.7.patch
          209 kB
          Phabricator
        25. ASF.LICENSE.NOT.GRANTED--D1521.7.patch
          209 kB
          Phabricator
        26. ASF.LICENSE.NOT.GRANTED--D1521.8.patch
          209 kB
          Phabricator
        27. ASF.LICENSE.NOT.GRANTED--D1521.8.patch
          209 kB
          Phabricator
        28. ASF.LICENSE.NOT.GRANTED--D1521.9.patch
          210 kB
          Phabricator
        29. ASF.LICENSE.NOT.GRANTED--D1521.9.patch
          210 kB
          Phabricator
        30. D1521.10.patch
          210 kB
          Michael Stack
        31. D1521.10.patch
          210 kB
          Michael Stack
        32. D1521.10.patch
          210 kB
          Michael Stack

        Issue Links

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: