Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14743

ArrayIndexOutOfBoundsException - HBASE-backed views' query with JOINs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: HBase Handler
    • Labels:
      None
    • Target Version/s:

      Description

      The stack:

      2016-09-13T09:38:49,972 ERROR [186b4545-65b5-4bfc-bc8e-3e14e251bb12 main] exec.Task: Job Submission failed with exception 'java.lang.ArrayIndexOutOfBoundsException(1)'
      java.lang.ArrayIndexOutOfBoundsException: 1
              at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.createFilterScan(HiveHBaseTableInputFormat.java:224)
              at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:492)
              at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:449)
              at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:370)
              at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:466)
              at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:356)
              at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:546)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:329)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:320)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
      
      

      Repro:

      CREATE TABLE HBASE_TABLE_TEST_1(
        cvalue string ,
        pk string,
       ccount int   )
      ROW FORMAT SERDE
        'org.apache.hadoop.hive.hbase.HBaseSerDe'
      STORED BY
        'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      WITH SERDEPROPERTIES (
        'hbase.columns.mapping'='cf:val,:key,cf2:count',
        'hbase.scan.cache'='500',
        'hbase.scan.cacheblocks'='false',
        'serialization.format'='1')
      TBLPROPERTIES (
        'hbase.table.name'='hbase_table_test_1',
        'serialization.null.format'=''  );
      
      
        CREATE VIEW VIEW_HBASE_TABLE_TEST_1 AS SELECT hbase_table_test_1.cvalue,hbase_table_test_1.pk,hbase_table_test_1.ccount FROM hbase_table_test_1 WHERE hbase_table_test_1.ccount IS NOT NULL;
      
      CREATE TABLE HBASE_TABLE_TEST_2(
        cvalue string ,
          pk string ,
         ccount int  )
      ROW FORMAT SERDE
        'org.apache.hadoop.hive.hbase.HBaseSerDe'
      STORED BY
        'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      WITH SERDEPROPERTIES (
        'hbase.columns.mapping'='cf:val,:key,cf2:count',
        'hbase.scan.cache'='500',
        'hbase.scan.cacheblocks'='false',
        'serialization.format'='1')
      TBLPROPERTIES (
        'hbase.table.name'='hbase_table_test_2',
        'serialization.null.format'='');
      
      
      CREATE VIEW VIEW_HBASE_TABLE_TEST_2 AS SELECT hbase_table_test_2.cvalue,hbase_table_test_2.pk,hbase_table_test_2.ccount FROM hbase_table_test_2 WHERE  hbase_table_test_2.pk >='3-0000h-0' AND hbase_table_test_2.pk <= '3-0000h-g' AND hbase_table_test_2.ccount IS NOT NULL;
      
      set hive.auto.convert.join=false;
      
        SELECT  p.cvalue cvalue
      FROM `VIEW_HBASE_TABLE_TEST_1` `p`
      LEFT OUTER JOIN `VIEW_HBASE_TABLE_TEST_2` `A1`
      ON `p`.cvalue = `A1`.cvalue
      LEFT OUTER JOIN `VIEW_HBASE_TABLE_TEST_1` `A2`
      ON `p`.cvalue = `A2`.cvalue;
      
      

        Attachments

        1. HIVE-14743.2.patch
          11 kB
          Yongzhi Chen
        2. HIVE-14743.1.patch
          11 kB
          Yongzhi Chen

          Activity

            People

            • Assignee:
              ychena Yongzhi Chen
              Reporter:
              ychena Yongzhi Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: