[IMPALA-3114] Add a pause monitor for impala processes.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Target Version: Impala 2.6.0
• Assignee: Bharath Vissapragada
• Reporter: Bharath Vissapragada
• Created: 02/Mar/16 13:12
• Updated: 16/Mar/16 09:46
• Resolved: 16/Mar/16 09:46

Description

In various scenarios we have seen impalad process getting blocked by OS for various reasons like swapping, hardware faults, system hangs etc. Typically the whole process gets stalled for a while and it is difficult to figure that out what happened based on logs. Logging such an event can give us an idea to correlate it with other events happening around that time. For example, we see something like this in the dmesg when the OS blocks it.

INFO: task impalad:21378 blocked for more than 120 seconds. 

This was originally observed in the hadoop daemons like namenode/datanode and a JVM pause monitor was implemented [1]. JVM had other issues like long GCs too and helped in such cases. Idea here is to implement something for Impala daemons too, to make debugging easier.

[1] https://github.com/cloudera/hadoop-common/blob/cdh5-2.3.0_5.1.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java