[DRILL-3579] Drill on Hive query fails if partition table has __HIVE_DEFAULT_PARTITION__

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Functions - Hive

Environment

Drill 1.1 on Hive 1.0

• Assignee: Rahul Kumar Challapalli
• Reporter: Hao Zhu
• Created: 30/Jul/15 00:25
• Updated: 03/Oct/15 00:08
• Resolved: 09/Aug/15 05:36

Description

If Hive's partition table has _HIVE_DEFAULT_PARTITION_ in the case of null values in the partition column, Drill on Hive query will fail.

Minimum reproduce:
1.Hive:

CREATE TABLE h1_testpart2(id INT) PARTITIONED BY(id2 int);
set hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE h1_testpart2 PARTITION(id2) SELECT 1 as id1 , 20150101 as id2 from h1_passwords limit 1;
INSERT OVERWRITE TABLE h1_testpart2 PARTITION(id2) SELECT 1 as id1 , null as id2 from h1_passwords limit 1;

2. Filesystem looks like:

h1 h1_testpart2]# ls -altr
total 2
drwxrwxrwx 89 mapr mapr 87 Jul 30 00:04 ..
drwxr-xr-x  2 mapr mapr  1 Jul 30 00:05 id2=20150101
drwxr-xr-x  2 mapr mapr  1 Jul 30 00:05 id2=__HIVE_DEFAULT_PARTITION__
drwxr-xr-x  4 mapr mapr  2 Jul 30 00:05 .

3.Drill will fail:

select * from h1_testpart2;
Error: SYSTEM ERROR: NumberFormatException: For input string: "__HIVE_DEFAULT_PARTITION__"

Fragment 0:0

[Error Id: 509eb392-db9a-42f3-96ea-fb597425f49f on h1.poc.com:31010]

  (java.lang.reflect.UndeclaredThrowableException) null
    org.apache.hadoop.security.UserGroupInformation.doAs():1581
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
    org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745
  Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while initializing HiveRecordReader: For input string: "__HIVE_DEFAULT_PARTITION__"
    org.apache.drill.exec.store.hive.HiveRecordReader.init():241
    org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
    org.apache.drill.exec.physical.impl.ImplCreator$2.run():138
    org.apache.drill.exec.physical.impl.ImplCreator$2.run():136
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1566
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
    org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745
  Caused By (java.lang.NumberFormatException) For input string: "__HIVE_DEFAULT_PARTITION__"
    java.lang.NumberFormatException.forInputString():65
    java.lang.Integer.parseInt():580
    java.lang.Integer.parseInt():615
    org.apache.drill.exec.store.hive.HiveRecordReader.convertPartitionType():605
    org.apache.drill.exec.store.hive.HiveRecordReader.init():236
    org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
    org.apache.drill.exec.physical.impl.ImplCreator$2.run():138
    org.apache.drill.exec.physical.impl.ImplCreator$2.run():136
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1566
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
    org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)

Issue Links

• links to: Review Board