[HIVE-11500] implement file footer / splits cache in HBase metastore

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Metastore
• Assignee: Sergey Shelukhin Assign to me
• Reporter: Sergey Shelukhin
• Created: 08/Aug/15 03:49
• Updated: 13/Oct/16 04:04

Description

We need to cache file metadata (e.g. ORC file footers) for split generation (which, on FSes that support fileId, will be valid permanently and only needs to be removed lazily when ORC file is erased or compacted), and potentially even some information about splits (e.g. grouping based on location that would be good for some short time), in HBase metastore.
It should be queryable by table. Partition predicate pushdown should be supported. If bucket pruning is added, that too. Given that we cannot cache file lists (we have to check FS for new/changed files anyway), and the difficulty of passing of data about partitions/etc. to split generation compared to paths, we will probably just filter by paths and fileIds. It might be different for splits

In later phases, it would be nice to save the (first category above) results of expensive work done by jobs, e.g. data size after decompression/decoding per column, etc. to avoid surprises when ORC encoding is very good, or very bad. Perhaps it can even be lazily generated. Here's a pony: 🐴

Issue Links

• incorporates: HIVE-14941 Renable stats_filemetadata.q test case
• incorporates: HIVE-12051 push file metadata PPD into HBase
• incorporates: HIVE-12052 automatically populate file metadata to HBase metastore based on config or table properties
• incorporates: HIVE-12801 document HIVE-11500 (metadata cache in HBase metastore)
• incorporates: HIVE-12925 make sure metastore footer cache doesn't get all functions
• incorporates: HIVE-11542 port fileId support on shims and splits from llap branch
• incorporates: HIVE-11552 implement basic methods for getting/putting file metadata
• incorporates: HIVE-11553 use basic file metadata cache in ETLSplitStrategy-related paths
• incorporates: HIVE-11595 refactor ORC footer reading to make it usable from outside
• incorporates: HIVE-11644 make sure Hive config is propagated to AM-side split generation
• incorporates: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy
• incorporates: HIVE-11676 implement metastore API to do file footer PPD
• incorporates: HIVE-11689 minor flow changes to ORC split generation
• incorporates: HIVE-11705 refactor SARG stripe filtering for ORC into a separate method
• incorporates: HIVE-11777 implement an option to have single ETL strategy for multiple directories
• incorporates: HIVE-11823 create a self-contained translation for SARG to be used by metastore
• incorporates: HIVE-11856 allow split strategies to run on threadpool
• incorporates: HIVE-12048 metastore file metadata cache should not be used when deltas are present
• incorporates: HIVE-12061 add file type support to file metadata by expr call
• incorporates: HIVE-12062 enable HBase metastore file metadata cache for tez tests
• incorporates: HIVE-12075 add analyze command to explictly cache file metadata in HBase metastore