Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7929

Impala query on HBASE table failing with InternalException: Required field*

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 3.2.0
    • Impala 3.2.0
    • Frontend
    • None
    • ghx-label-7

    Description

      This looks a corner case bug demonstrated at impala-hbase boundary.

      The way to reproduce:

      Create a table in hive shell,

      create database abc;
      
      CREATE TABLE abc.test_hbase1 (k STRING, c STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ('
      hbase.columns.mapping'=':key,cf:c', 'serialization.format'='1') TBLPROPERTIES ('hbase.table.name'='test_hbase1', 'storage_handler'='o
      rg.apache.hadoop.hive.hbase.HBaseStorageHandler');
      
      

      Then issue query at impala shell:

      select * from abc.test_hbase1 where k != "row1"; 
      

      Observe:

      Query: select * from abc.test_hbase1 where k != "row1"                                     
      Query submitted at: 2018-12-04 17:02:42 (Coordinator: http://xyz:25000)    
      ERROR: InternalException: Required field 'qualifier' was not present! Struct: THBaseFilter(family::key, qualifier:null, op_ordinal:3, filter_constant:row1)
      

      More observations:

      1. Replacing k != "row1" with k <> "row1" fails the same way. However, replacing it with other operators, such as ">", "<", "=", all works.
      2. Replacing k != "row1 with c != "row1", it succeeded without the error reported above.

      The above example uses a two-column table, creating a similar table with three columns fails the same way: adding inequality predicate on the first column fails, adding inequility predicate doesn't fail.

      The code that issues the error message is in HBase, it seems Impala did not pass the needed info to HBase in this special case. Also wonder if it's because the first column of the table is the key in hbase table that could reveal the bug.

      hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java:      throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
      hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java:      throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
      hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
      hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
      hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java:        throw new org.apache.thrift.protocol.TProtocolException("Required field 'qualifier' was not present! Struct: " + toString());
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yzhangal Yongjun Zhang
            yzhangal Yongjun Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment