[IMPALA-6500] Impala crashes under certain hypervisors that return out-of-range CPU IDs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Backend
• Labels: crash
• Environment: RHEL 6.5 (Santiago), Kernel version: 2.6.32-431.el6.x86_64, CDH 5.13.1 single node, Impala 2.10, VMware ESXi, v5.0.0
• Target Version: Impala 3.0, Impala 2.12.0
• Epic Color: ghx-label-9
• Assignee: Tim Armstrong
• Reporter: Osama Suleiman
• Created: 12/Feb/18 12:39
• Updated: 20/Jun/18 19:09
• Resolved: 09/Mar/18 17:00

Description

I have a Parquet table created by Hive and I am doing multiple different queries on it, such as:

SELECT product_category,
SUM(cast(profit AS DECIMAL(15,2))) as total_profit,
SUM(cast(sales AS DECIMAL(15,2))) as total_sales
FROM copy_orders
GROUP BY product_category;

and:

SELECT customer_name,
SUM(cast(profit AS DECIMAL(15,2))) as total_profit,
SUM(cast(sales AS DECIMAL(15,2))) as total_sales
FROM copy_orders
GROUP BY customer_name
ORDER BY total_profit DESC
LIMIT 10;

These two queries tend to run successfully in some rare occasions, most of the time running those queries on HUE's Impala query editor will return:

Could not connect to hostname:21050 (code THRIFTTRANSPORT): TTransportException('Could not connect to hostname:21050',)

Simultaneously, the Impala Daemon crashes according to the Cloudera Manager and then it will work again approximately 1 min later. Meanwhile, You can run other simple queries and it will run successfully.

I have attached a log file for a sample run of one of the queries since they all generate relevant logs. I have tried to use SET disable_codegen=1 but the problem resumed.

I have added both the impalad.ERROR and impalad.INFO files after running the 2nd query 2 times: the 1st time I used "SET disable_codegen=1" which sometimes work, and after that, it ran successfully (represented by impalad.INFO). The other time, at which the query has failed, is logged in the (impalad.INFO2) while impalad.ERROR doesn't seem to have changed at all. It seems that all the old logs gets removed from the main logging files (impalad.INFO and impalad.ERROR) since running the query keeps restarting the Impala Daemon.