[IMPALA-6530] Track time spent opening HDFS file handles

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0, Impala 2.11.0
• Fix Version/s: Impala 2.12.0
• Component/s: Backend
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-2
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 15/Feb/18 23:39
• Updated: 28/Feb/18 16:19
• Resolved: 28/Feb/18 16:19

Description

When the HDFS NameNode is overloaded, opening an HDFS file handle can become a time consuming operation and can become the bottleneck for the system. Currently, there is no statistic tracking time spent opening file handles. This limits the ability to determine the extent of the HDFS NameNode problem, especially when given only a single query profile.

Impala should have a statistic at the HDFS scan node level tracking time spent opening file handles.