Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4304

Select from hive hbase table fails with NPE when hbase.table.name is not used

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      git.commit.id.abbrev=3d0b4b0

      I can query against the following hive/hbase table fine:

      create table hbase_voter(key int,name varchar(30),age int,registration string,contributions decimal(6,2),voternum smallint,create_date timestamp)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ':key,onecf:name,twocf:age,twocf:registration,threecf:contributions,threecf:voterzone,fourcf:create_date') TBLPROPERTIES("hbase.table.name" = "hbase_voter");

      However, if I omit the "hbase.table.name" (which is optional in hive), it fails when I query against this table from sqlline:

      create table hbase_voter(key int,name varchar(30),age int,registration string,contributions decimal(6,2),voternum smallint,create_date timestamp)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ':key,onecf:name,twocf:age,twocf:registration,threecf:contributions,threecf:voterzone,fourcf:create_date');

      Here is the stack trace:

      SYSTEM ERROR: NullPointerException
      
      
      
        (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Internal error: Error while applying rule DrillPushProjIntoScan, args [rel#8:LogicalProject.NONE.ANY([]).[](input=rel#7:Subset#0.ENUMERABLE.ANY([]).[],key=$0,name=$1,age=$2,registration=$3,contributions=$4,voternum=$5,create_date=$6), rel#2:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[hive, hbase_voter2])]
          org.apache.drill.exec.work.foreman.Foreman.run():261
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.AssertionError) Internal error: Error while applying rule DrillPushProjIntoScan, args [rel#8:LogicalProject.NONE.ANY([]).[](input=rel#7:Subset#0.ENUMERABLE.ANY([]).[],key=$0,name=$1,age=$2,registration=$3,contributions=$4,voternum=$5,create_date=$6), rel#2:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[hive, hbase_voter2])]
          org.apache.calcite.util.Util.newInternal():792
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.AssertionError) Internal error: Error occurred while applying rule DrillPushProjIntoScan
          org.apache.calcite.util.Util.newInternal():792
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():150
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.reflect.UndeclaredThrowableException) null
          com.sun.proxy.$Proxy65.getNonCumulativeCost():-1
          org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost():115
          org.apache.calcite.plan.volcano.VolcanoPlanner.getCost():1112
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0():363
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements():344
          org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet():1827
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1760
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():117
          org.apache.calcite.rel.AbstractRelNode.onRegister():305
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1658
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1940
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():138
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.reflect.InvocationTargetException) null
          sun.reflect.NativeMethodAccessorImpl.invoke0():-2
          sun.reflect.NativeMethodAccessorImpl.invoke():57
          sun.reflect.DelegatingMethodAccessorImpl.invoke():43
          java.lang.reflect.Method.invoke():606
          org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():182
          com.sun.proxy.$Proxy65.getNonCumulativeCost():-1
          org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost():115
          org.apache.calcite.plan.volcano.VolcanoPlanner.getCost():1112
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0():363
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements():344
          org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet():1827
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1760
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():117
          org.apache.calcite.rel.AbstractRelNode.onRegister():305
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1658
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1940
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():138
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) java.io.IOException: Failed to get numRows from HiveTable
          org.apache.drill.exec.store.hive.HiveScan.getScanStats():233
          org.apache.drill.exec.physical.base.AbstractGroupScan.getScanStats():76
          org.apache.drill.exec.planner.logical.DrillScanRel.computeSelfCost():164
          org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost():165
          sun.reflect.NativeMethodAccessorImpl.invoke0():-2
          sun.reflect.NativeMethodAccessorImpl.invoke():57
          sun.reflect.DelegatingMethodAccessorImpl.invoke():43
          java.lang.reflect.Method.invoke():606
          org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():182
          com.sun.proxy.$Proxy65.getNonCumulativeCost():-1
          org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost():115
          org.apache.calcite.plan.volcano.VolcanoPlanner.getCost():1112
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0():363
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements():344
          org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet():1827
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1760
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():117
          org.apache.calcite.rel.AbstractRelNode.onRegister():305
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1658
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1940
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():138
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.io.IOException) Failed to get numRows from HiveTable
          org.apache.drill.exec.store.hive.HiveMetadataProvider.getStats():110
          org.apache.drill.exec.store.hive.HiveScan.getScanStats():224
          org.apache.drill.exec.physical.base.AbstractGroupScan.getScanStats():76
          org.apache.drill.exec.planner.logical.DrillScanRel.computeSelfCost():164
          org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost():165
          sun.reflect.NativeMethodAccessorImpl.invoke0():-2
          sun.reflect.NativeMethodAccessorImpl.invoke():57
          sun.reflect.DelegatingMethodAccessorImpl.invoke():43
          java.lang.reflect.Method.invoke():606
          org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():182
          com.sun.proxy.$Proxy65.getNonCumulativeCost():-1
          org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost():115
          org.apache.calcite.plan.volcano.VolcanoPlanner.getCost():1112
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0():363
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements():344
          org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet():1827
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1760
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():117
          org.apache.calcite.rel.AbstractRelNode.onRegister():305
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1658
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1940
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():138
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
        Caused By (java.lang.NullPointerException) null
          org.apache.hadoop.hbase.util.Bytes.toBytes():503
          org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal():444
          org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits():432
          org.apache.drill.exec.store.hive.HiveMetadataProvider$1.run():250
          org.apache.drill.exec.store.hive.HiveMetadataProvider$1.run():238
          java.security.AccessController.doPrivileged():-2
          javax.security.auth.Subject.doAs():415
          org.apache.hadoop.security.UserGroupInformation.doAs():1595
          org.apache.drill.exec.store.hive.HiveMetadataProvider.splitInputWithUGI():238
          org.apache.drill.exec.store.hive.HiveMetadataProvider.getTableInputSplits():125
          org.apache.drill.exec.store.hive.HiveMetadataProvider.getStats():93
          org.apache.drill.exec.store.hive.HiveScan.getScanStats():224
          org.apache.drill.exec.physical.base.AbstractGroupScan.getScanStats():76
          org.apache.drill.exec.planner.logical.DrillScanRel.computeSelfCost():164
          org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost():165
          sun.reflect.NativeMethodAccessorImpl.invoke0():-2
          sun.reflect.NativeMethodAccessorImpl.invoke():57
          sun.reflect.DelegatingMethodAccessorImpl.invoke():43
          java.lang.reflect.Method.invoke():606
          org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke():182
          com.sun.proxy.$Proxy65.getNonCumulativeCost():-1
          org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost():115
          org.apache.calcite.plan.volcano.VolcanoPlanner.getCost():1112
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements0():363
          org.apache.calcite.plan.volcano.RelSubset.propagateCostImprovements():344
          org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet():1827
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1760
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():117
          org.apache.calcite.rel.AbstractRelNode.onRegister():305
          org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl():1658
          org.apache.calcite.plan.volcano.VolcanoPlanner.register():1017
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1037
          org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered():1940
          org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():138
          org.apache.calcite.plan.RelOptRuleCall.transformTo():213
          org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():86
          org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
          org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
          org.apache.calcite.tools.Programs$RuleSetProgram.run():303
          org.apache.calcite.prepare.PlannerImpl.transform():313
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning():542
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():218
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():252
          org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
          org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():199
          org.apache.drill.exec.work.foreman.Foreman.runSQL():924
          org.apache.drill.exec.work.foreman.Foreman.run():250
          java.util.concurrent.ThreadPoolExecutor.runWorker():1145
          java.util.concurrent.ThreadPoolExecutor$Worker.run():615
          java.lang.Thread.run():745
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              knguyen Krystal
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: