Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9452

Use HBase to store Hive metadata

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: hbase-metastore-branch
    • Fix Version/s: None
    • Component/s: Metastore
    • Labels:
      None

      Description

      qThis is an umbrella JIRA for a project to explore using HBase to store the Hive data catalog (ie the metastore). This project has several goals:

      1. The current metastore implementation is slow when tables have thousands or more partitions. With Tez and Spark engines we are pushing Hive to a point where queries only take a few seconds to run. But planning the query can take as long as running it. Much of this time is spent in metadata operations.
      2. Due to scale limitations we have never allowed tasks to communicate directly with the metastore. However, with the development of LLAP this requirement will have to be relaxed. If we can relax this there are other use cases that could benefit from this.
      3. Eating our own dogfood. Rather than using external systems to store our metadata there are benefits to using other components in the Hadoop system.

      The proposal is to create a new branch and work on the prototype there.

        Attachments

        1. HBaseMetastoreApproach.pdf
          945 kB
          Alan Gates

          Issue Links

            Activity

              People

              • Assignee:
                gates Alan Gates
                Reporter:
                gates Alan Gates
              • Votes:
                8 Vote for this issue
                Watchers:
                49 Start watching this issue

                Dates

                • Created:
                  Updated: