Type: New Feature
Affects Version/s: None
Fix Version/s: None
Hive had an implementation of indexing that was made somewhat obsolete given the introduction of columnar file formats with their own internal indexing.
I propose that Hive introduce Indexing again.
- Column Index: Stored in HBase
- Full-Text Index: Stored in Solr
The basic idea is that, the key in HBase is the record and the value is the relative file path of the data in the Hive table.
Performing an INSERT statement creates the index for each record.
When generating the explain plan, only the files involved in the query are considered.
This would prevents having to scan large amounts of data for the typical BI tools when the set of data is known to be very small.