Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15240 Go Big BucketCache Fixes
  3. HBASE-15477

Do not save 'next block header' when we cache hfileblocks

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: BlockCache, Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Fix over-persisting in blockcache; no longer save the block PLUS the header of the next block (33 bytes) when writing the cache.

      Also removes support for hfileblock v1; hfile block v1 was used writing hfile v1. hfile v1 was the default in hbase before hbase-0.92. hbase.96 would not start unless all v1 hfiles had been compacted out of the cluster.
      Show
      Fix over-persisting in blockcache; no longer save the block PLUS the header of the next block (33 bytes) when writing the cache. Also removes support for hfileblock v1; hfile block v1 was used writing hfile v1. hfile v1 was the default in hbase before hbase-0.92. hbase.96 would not start unless all v1 hfiles had been compacted out of the cluster.

      Description

      When we read from HDFS, we overread to pick up the next blocks header.
      Doing this saves a seek as we move through the hfile; we save having to
      do an explicit seek just to read the block header every time we need to
      read the body. We used to read in the next header as part of the
      current blocks buffer. This buffer was then what got persisted to
      blockcache; so we were over-persisting wrtiting out our block plus the
      next blocks' header (overpersisting 33 bytes). Parse of HFileBlock
      complicated by this extra tail. Fix.

        Attachments

        1. 15366v4.patch
          145 kB
          Michael Stack
        2. 15477.backport.branch-1.patch
          141 kB
          Michael Stack
        3. 15477.backport.branch-1.v2.patch
          118 kB
          Michael Stack
        4. 15477.backport.branch-1.v3.patch
          146 kB
          Michael Stack
        5. 15477.backport.branch-1.v4.patch
          148 kB
          Michael Stack
        6. 15477.backport.branch-1.v4.patch
          148 kB
          Michael Stack
        7. 15477.backport.branch-1.v5.patch
          148 kB
          Michael Stack
        8. 15477.backport.branch-1.v5.patch
          148 kB
          Michael Stack
        9. 15477.backport.branch-1.v6.patch
          150 kB
          Michael Stack
        10. 15477.backport.branch-1.v7.patch
          180 kB
          Michael Stack
        11. 15477.patch
          139 kB
          Michael Stack
        12. 15477v2.patch
          142 kB
          Michael Stack
        13. 15477v3.patch
          145 kB
          Michael Stack
        14. 15477v3.patch
          145 kB
          Michael Stack
        15. 15477v4.patch
          145 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: