Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11397

When merging expired stripes, we need to create an empty file to preserve metadata.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.2
    • Fix Version/s: 0.99.0, 0.98.4
    • Component/s: Compaction
    • Labels:
      None
    • Environment:

      jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2

    • Hadoop Flags:
      Reviewed

      Description

      Stripe Compaction is a good feature in 0.96 and 0.98. But when I used it in a heavy-write non-uniform row keys scenario(e.g. time dimension in a key), I came across some problems.
      I made my stripes split at the size of 2G(hbase.store.stripe.sizeToSplit=2G), and soon there were tens of them. It was true that only the last stripe receiving the new keys kept compacting - old data didn't compact as much, or at all. However, the old stripes were still there when they all expired. I checked the source code and found that when compacting expired stripes, the StoreScanner may return no KVs so that SizeMultiWriter.append() is never called. That's to say, NO NEW FILE WILL BE CREATED.
      My solution is to create an empty file to preserve metadata at the end of the SizeMultiWriter.commitWritersInternal().

        Attachments

        1. HBASE-11397-v2.patch
          1 kB
          Victor Xu
        2. HBASE-11397-test.patch
          8 kB
          Victor Xu
        3. HBASE-11397-Stripe-Info.png
          158 kB
          Victor Xu
        4. HBASE-11397-RS-Log.png
          904 kB
          Victor Xu
        5. HBASE-11397-HDFS.png
          324 kB
          Victor Xu
        6. HBASE-11397-AssertionError.png
          236 kB
          Victor Xu
        7. HBASE-11397.patch
          1 kB
          Victor Xu
        8. 11397-v3.patch
          9 kB
          Ted Yu

          Activity

            People

            • Assignee:
              victorunique Victor Xu
              Reporter:
              victorunique Victor Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: