[HIVE-9452] Use HBase to store Hive metadata

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: hbase-metastore-branch
• Component/s: Metastore
• Assignee: Alan Gates
• Reporter: Alan Gates
• Created: 23/Jan/15 18:41
• Updated: 30/May/22 19:10

Description

qThis is an umbrella JIRA for a project to explore using HBase to store the Hive data catalog (ie the metastore). This project has several goals:

1. The current metastore implementation is slow when tables have thousands or more partitions. With Tez and Spark engines we are pushing Hive to a point where queries only take a few seconds to run. But planning the query can take as long as running it. Much of this time is spent in metadata operations.
2. Due to scale limitations we have never allowed tasks to communicate directly with the metastore. However, with the development of LLAP this requirement will have to be relaxed. If we can relax this there are other use cases that could benefit from this.
3. Eating our own dogfood. Rather than using external systems to store our metadata there are benefits to using other components in the Hadoop system.

The proposal is to create a new branch and work on the prototype there.

Issue Links

• is broken by: HIVE-11085 Alter table fail with NPE if schema change
• is related to: HIVE-17234 Remove HBase metastore from master
• relates to: HIVE-17234 Remove HBase metastore from master

Sub-Tasks

• Initial patch [hbase-metastore branch]: Closed
• Add support for getDatabases and alterDatabase calls [hbase-metastore branch]: Closed
• Support all get tables [hbase-metastore branch]: Closed
• Need a tool to export metadata from RDBMS based metastore into HBase: Closed
• Fill out remaining partition functions in HBaseStore: Closed
• Implement privileges call in HBaseStore: Closed
• Introduce a stats cache for aggregate stats in HBase metastore [hbase-metastore branch]: Closed
• Need to add indices and privileges to HBaseImport and HBaseSchemaTool [hbase-metastore branch]: Open
• Documentation for HBase metastore: Open
• Remove M* classes from RawStore interface: Closed
• Move serialization of objects in HBase to protocol buffers: Closed
• Refactor HBaseReadWrite to allow different implementations underneath: Closed
• Need a way to record time spent in various metastore functions: Open
• Perform stats aggregation in HBase co-processor [hbase-metastore branch]: Open
• Investigate ways to improve NDV calculations during stats aggregation [hbase-metastore branch]: Resolved
• Implement functions methods in HBaseStore [hbase-metastore branch]: Closed
• Add connection manager for Tephra: Closed
• Generate Hbase execution plan for partition filter conditions in HbaseStore api calls - initial changes: Closed
• Optimize handling of partition condition expression that result in invalid filter string: Closed
• get metatool to work with hbase metastore: Open
• Fix test failure in TestAggregateStatsCache: Resolved
• Support filter on non-first partition key and non-string partition key: Closed
• Add tests for partition level statistics + refactor stats tests of TestHBaseStore [hbase-metastore branch]: Resolved
• Unit test against HBase Metastore: Closed
• Describe a non-partitioned table fail: Closed
• Get partial stats instead of complete stats in some queries: Closed
• Fix Unit test failures when HBase Metastore is used: Open
• Fix alter related Unit tests for HBase metastore: Open
• Fix stats related unit tests for HBase metastore: Open
• Fix partitions related unit tests for HBase metastore: Open
• Invalidate aggregate column stats on alter partition: Closed
• implement file footer / splits cache in HBase metastore: Open
• merge master into branch: Closed
• merge master into branch: Closed
• Fix TestMiniTezCliDriver test failures when HBase Metastore is used: Closed
• import tool is too brittle (fails on existing role): Open
• import tool should print help by default: Closed
• import tool fails on non-secure cluster: Open
• NPE in stats conversion with HBase metastore: Closed
• Fix UT regressions on hbase-metastore branch: Closed
• Exclude hbase-metastore for hadoop-1: Closed
• Merge hbase-metastore branch to trunk: Closed
• HBaseImport should import basic stats and column stats: Open
• Exclude hbase-metastore in itests for hadoop-1: Closed
• HBase Port conflict for MiniHBaseCluster: Closed
• Renable stats_filemetadata.q test case: Open