[IMPALA-609] Impala should provide a more accurate error message when region server is unhealthy - fails with NoClassDefFoundError

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 1.1
• Component/s: Frontend
• Labels: impala
• Environment: CentOS release 6.4 (Final); CDH: CDH 4.4.0-1.cdh4.4.0.p0.39; JDK: java version "1.7.0_21"
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Tony Xu
• Created: 01/Oct/13 17:14
• Updated: 16/Sep/19 22:38
• Resolved: 16/Sep/19 22:38

Description

When running query in Impala (Command line) "select name from product where manufacturername = "Cat's Pride" limit 1;", Impala returns error message: "org.apache.hadoop.ipc.RemoteException(java.lang.NoClassDefFoundError): IPC server unable to read call parameters: Could not initialize class org.apache.hadoop.hbase.util.Classes"

Note:
1. Basic "select * from table_name limit 1;" works in both command line and Hue Impala UI.
2. We used CM to install Impala, "All Services" -> "Actions" -> "Add a Service" -> Then choose Impala for the initial installation then updated through "Parcels" interface.
3. The same query works in hive.

Cause:
One of the "region server" couldn't talk to the Hbase master, there are also error messages in Hbase log file on the problematic node. The error message I caught from Hbase log is:
=================================
WARN org.apache.hadoop.ipc.HBaseServer: Unable to read call parameters for client 10.6.70.3.
=================================

10.6.70.3 is the Hbase master's IP address.

Impala should have failed with a better error message than "NoClassDefFoundError".

Impala log:
http://paste.ubuntu.com/6179531/

Hbase unhealthy canary report log:
http://paste.ubuntu.com/6179569/