[IMPALA-4840] Fix REFRESH perf issues.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Catalog
• Labels: catalog-server performance
• Assignee: Bharath Vissapragada
• Reporter: Bharath Vissapragada
• Created: 30/Jan/17 05:49
• Updated: 16/Feb/17 07:34
• Resolved: 16/Feb/17 07:32

Description

IMPALA-4172 changed the behavior of "REFRESH" by making it to use a new code path that reloads the file block metadata of the entire table rather than re-using the metadata of already loaded partitions. This was done under the assumption that the actual metadata loading was sped up so much that we'd have very little impact even if we re-compute the block metadata of already loaded files. Based on our testing, this clearly isn't the case, especially if the table is huge. We need to revert this behavior.

Issue Links

• blocks: IMPALA-4943 Improve metadata load speed for "alter table add partition".
• is related to: IMPALA-4172 Switch from using getFileBlockLocations to BlockLocation methods (Potential 50% speedup in metadata loading)