Description
Need to take a look at the best flow. It won't be much different if we do filtering metastore call for each partition. So perhaps we'd need the custom sync point/batching after all.
Or we can make it opportunistic and not fetch any footers unless it can be pushed down to metastore or fetched from local cache, that way the only slow threaded op is directory listings
Attachments
Attachments
Issue Links
- blocks
-
HIVE-12925 make sure metastore footer cache doesn't get all functions
- Open
- is blocked by
-
HIVE-11676 implement metastore API to do file footer PPD
- Closed
-
HIVE-11777 implement an option to have single ETL strategy for multiple directories
- Closed
-
HIVE-11856 allow split strategies to run on threadpool
- Closed
- is part of
-
HIVE-11500 implement file footer / splits cache in HBase metastore
- Open
- links to