[HIVE-12052] automatically populate file metadata to HBase metastore based on config or table properties

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 07/Oct/15 01:12
• Updated: 06/Jul/16 01:29

Description

As discussed in HIVE-11500
Should use a table property similar to auto.purge.
Then, when this setting is set, partitions are added (convertToMPart is a good source to find all the paths for that), after compactions, after load/non-ACID insert, and periodically (configurable), the storage locations should be scanned for new files and cache updated accordingly. All the updates should probably be in the background thread and taken from queue (high pri from most ops, low pri from enabling the property and from periodic updates) to avoid high load on HDFS from metastore.

Issue Links

• is blocked by: HIVE-12075 add analyze command to explictly cache file metadata in HBase metastore
• is part of: HIVE-11500 implement file footer / splits cache in HBase metastore