[IMPALA-7929] Impala query on HBASE table failing with InternalException: Required field*

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.2.0
• Fix Version/s: Impala 3.2.0
• Component/s: Frontend
• Epic Color: ghx-label-7
• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 05/Dec/18 23:00
• Updated: 08/Feb/19 01:21
• Resolved: 08/Feb/19 01:21

Description

This looks a corner case bug demonstrated at impala-hbase boundary.

The way to reproduce:

Create a table in hive shell,

create database abc;

CREATE TABLE abc.test_hbase1 (k STRING, c STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ('
hbase.columns.mapping'=':key,cf:c', 'serialization.format'='1') TBLPROPERTIES ('hbase.table.name'='test_hbase1', 'storage_handler'='o
rg.apache.hadoop.hive.hbase.HBaseStorageHandler');

Then issue query at impala shell:

select * from abc.test_hbase1 where k != "row1"; 

Observe:

Query: select * from abc.test_hbase1 where k != "row1"                                     
Query submitted at: 2018-12-04 17:02:42 (Coordinator: http://xyz:25000)    
ERROR: InternalException: Required field 'qualifier' was not present! Struct: THBaseFilter(family::key, qualifier:null, op_ordinal:3, filter_constant:row1)

More observations:

1. Replacing k != "row1" with k <> "row1" fails the same way. However, replacing it with other operators, such as ">", "<", "=", all works.
2. Replacing k != "row1 with c != "row1", it succeeded without the error reported above.

The above example uses a two-column table, creating a similar table with three columns fails the same way: adding inequality predicate on the first column fails, adding inequility predicate doesn't fail.

The code that issues the error message is in HBase, it seems Impala did not pass the needed info to HBase in this special case. Also wonder if it's because the first column of the table is the key in hbase table that could reveal the bug.

hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java:      throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java:      throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());