Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20636

Introduce two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0
    • Component/s: HFile, regionserver, Scanners
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED
      1. ROWPREFIX_FIXED_LENGTH: specify the length of the prefix
      2. ROWPREFIX_DELIMITED: specify the delimiter of the prefix
      Need to specify parameters for these two types of bloomfilter, otherwise the table will fail to create
      Example:
      create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_FIXED_LENGTH', CONFIGURATION => {'RowPrefixBloomFilter.prefix_length' => '10'}}
      create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_DELIMITED', CONFIGURATION => {'RowPrefixDelimitedBloomFilter.delimiter' => '#'}}
      Show
      Add two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED 1. ROWPREFIX_FIXED_LENGTH: specify the length of the prefix 2. ROWPREFIX_DELIMITED: specify the delimiter of the prefix Need to specify parameters for these two types of bloomfilter, otherwise the table will fail to create Example: create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_FIXED_LENGTH', CONFIGURATION => {'RowPrefixBloomFilter.prefix_length' => '10'}} create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_DELIMITED', CONFIGURATION => {'RowPrefixDelimitedBloomFilter.delimiter' => '#'}}

      Description

      As we all know, HBase uses BloomFilter(ROW and ROWCOL) to filter unnecessary files to improve read performance. But they only support Get and do not support Scan.

      In our company(Tencent), many users need to scan all rows with the same prefix, such as Tencent Game. Game user's some operational record will be written into HBase, each game user will have a lot of records, the rowkey is constructed as userid+'#'+timestamps. So we can scan all records for a given user for a specified period.

      For this scenario, we designed the prefix Bloom filter. If the startRow and stopRow of the Scan has a valid common prefix, the scan will be allowed to use BloomFilter to filter files which will enhance the performance of the scan.

      Now, this feature has been running on our cluster over a year, and scan performance for this scenario has been improved by more than one times than before.

        Attachments

        1. HBASE-20636.master.005.patch
          65 kB
          Andrew Purtell
        2. HBASE-20636.master.004.patch
          65 kB
          Andrew Purtell
        3. HBASE-20636.master.003.patch
          65 kB
          Guangxu Cheng
        4. HBASE-20636.master.002.patch
          59 kB
          Guangxu Cheng
        5. HBASE-20636.master.001.patch
          58 kB
          Guangxu Cheng

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                andrewcheng Guangxu Cheng
                Reporter:
                andrewcheng Guangxu Cheng
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: