HBase
  1. HBase
  2. HBASE-2037

Alternate indexed hbase implementation; speeds scans by adding indexes to regions rather secondary tables

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.3
    • Component/s: Coprocessors
    • Labels:
      None

      Description

      Purpose

      The goal of the indexed HBase contrib is to speed up scans by indexing HBase columns. Indexed HBase (IHbase) is different from the indexed tables in transactional HBase (ITHbase): while the indexes in ITHBase are, in fact, hbase tables using the indexed column's values as row keys, IHbase creates indexes at the region level. The differences are summarized in below.

      + global ordering
      ITHBase: yes
      IHBase: no
      Comment: IHBase has an index for each region. The flip side of not having global ordering is compatibility with the good old HRegion: results are coming back in row order (and not value order as in THBase)

      + Full table scan?
      ITHBase: no
      IHBase: no
      Comment: ITHbase does a partial scan on the index table. IHbase supports specifying start/end rows to limit the number of scanned regions

      + Multiple Index Usage
      ITHBase: no
      IHBase: yes
      Comment: IHBase can take advantage of multiple indexes in the same scan. IHBase IdxScan object accepts an Expression which allows intersection/ unison of several indexed
      column criteria

      + Extra disk storage
      ITHBase: yes
      IHBase: no
      Comment: IHbase indexes are created when the region starts/flushes and do not require any extra storage

      + Extra RAM
      ITHBase: yes
      IHBase: yes
      Comment: IHbase indexes are in memory and hence increase the memory overhead. THbase indexes increase the number of regions each region server has to support thus costing memory too

      + Parallel scanning support
      ITHBase: no
      IHBase: yes
      In ITHbase the index table needs to be consulted and then GETs are issued for each matching row. The behavior of IHBase (as perceived by the client) is no different than a regular scan and hence supports parallel scanning seamlessly. parallel GET can be implemented to speedup ITHbase scans

      Why IHbase should outperform ITHBase
      1. More flexible: a. Supports range queries and multi-index queries b. Supports different types - not only byte arrays
      2. Less overhead: ITHbase pays at least two 'table roundtrips' - one for the index table and the other for the main table
      3. Quicker index expression evaluation: IHBase is using dedicated index data structures while ITHbase is using the regular HRegion scan facilities

      Implementation notes
      • Only index Storefiles.Every index scan performs a full memstore scan. Indexing the memstore will be implemented only if scanning the memstore will prove to be a performance bottleneck
      • Index expression evaluation is performed using bit sets.There are two types of bitsets: compressed and expanded. An index will typically store a compressed bitset while an expression evaluator will most probably use an expanded bitset
      + TODO

      This patch changes some some of hbase core so can instantiate other than default HRegion. Fixes bugs in filter too.

      Would like to add this as a contrib. package on 0.20 branch in time for 0.20.3 if possible.

      1. index.html
        21 kB
        stack
      2. idx-hbase3.patch
        2.53 MB
        stack
      3. idx-hbase2.patch
        2.46 MB
        stack

        Issue Links

          Activity

          stack created issue -
          stack made changes -
          Field Original Value New Value
          Attachment idx-hbase2.patch [ 12427625 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-2038 [ HBASE-2038 ]
          stack made changes -
          Attachment idx-hbase3.patch [ 12429425 ]
          stack made changes -
          Attachment index.html [ 12429426 ]
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jean-Daniel Cryans made changes -
          Assignee stack [ stack ]
          Todd Lipcon made changes -
          Component/s coprocessors [ 12314191 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              stack
              Reporter:
              stack
            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development