Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3529

Add search to HBase

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Later
    • 0.90.0
    • None
    • None
    • None

    Description

      Using the Apache Lucene library we can add freetext search to HBase. The advantages of this are:

      • HBase is highly scalable and distributed
      • HBase is realtime
      • Lucene is a fast inverted index and will soon be realtime (see LUCENE-2312)
      • Lucene offers many types of queries not currently available in HBase (eg, AND, OR, NOT, phrase, etc)
      • It's easier to build scalable realtime systems on top of already architecturally sound, scalable realtime data system, eg, HBase.
      • Scaling realtime search will be as simple as scaling HBase.

      Phase 1 - Indexing:

      • Integrate Lucene into HBase such that an index mirrors a given region. This means cascading add, update, and deletes between a Lucene index and an HBase region (and vice versa).
      • Define meta-data to mark a region as indexed, and use a Solr schema to allow the user to define the fields and analyzers.
      • Integrate with the HLog to ensure that index recovery can occur properly (eg, on region server failure)
      • Mirror region splits with indexes (use Lucene's IndexSplitter?)
      • When a region is written to HDFS, also write the corresponding Lucene index to HDFS.
      • A row key will be the ID of a given Lucene document. The Lucene docstore will explicitly not be used because the document/row data is stored in HBase. We will need to solve what the best data structure for efficiently mapping a docid -> row key is. It could be a docstore, field cache, column stride fields, or some other mechanism.
      • Write unit tests for the above

      Phase 2 - Queries:

      • Enable distributed Lucene queries
      • Regions that have Lucene indexes are inherently available and may be searched on, meaning there's no need for a separate search related system in Zookeeper.
      • Integrate search with HBase's RPC mechanis

      Attachments

        1. HBASE-3529.patch
          41 kB
          Jason Rutherglen
        2. HDFS-APPEND-0.20-LOCAL-FILE.patch
          8 kB
          Jason Rutherglen

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-2919 IndexSplitter that divides by primary key term

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-3296 Enable passing a config into PKIndexSplitter

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-3257 Coprocessors: Extend server side integration API to include HLog operations

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SOLR-2563 Allow generic pluggable file system implementations

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6311 Add support for unix domain sockets to JNI libs

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HBASE-883 Secondary Indexes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-3786 Enhance MasterCoprocessorHost to include notification of balancing of each region

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-4048 [Coprocessors] Support configuration of coprocessor at load time

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-347 DFS read performance suboptimal when client co-located on nodes with data

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-941 Datanode xceiver protocol should allow reuse of a connection

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-1431 CommComponent abstracted

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-2565 Prevent IW#close and cut over to IW#commit

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-3810 Registering a Coprocessor at HTableDescriptor should be less strict

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              Unassigned Unassigned
              jasonrutherglen Jason Rutherglen
              Votes:
              37 Vote for this issue
              Watchers:
              87 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: