Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3579

Drill on Hive query fails if partition table has __HIVE_DEFAULT_PARTITION__

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: Functions - Hive
    • Labels:
      None
    • Environment:

      Drill 1.1 on Hive 1.0

      Description

      If Hive's partition table has _HIVE_DEFAULT_PARTITION_ in the case of null values in the partition column, Drill on Hive query will fail.

      Minimum reproduce:
      1.Hive:

      CREATE TABLE h1_testpart2(id INT) PARTITIONED BY(id2 int);
      set hive.exec.dynamic.partition.mode=nonstrict;
      INSERT OVERWRITE TABLE h1_testpart2 PARTITION(id2) SELECT 1 as id1 , 20150101 as id2 from h1_passwords limit 1;
      INSERT OVERWRITE TABLE h1_testpart2 PARTITION(id2) SELECT 1 as id1 , null as id2 from h1_passwords limit 1;
      
      

      2. Filesystem looks like:

      h1 h1_testpart2]# ls -altr
      total 2
      drwxrwxrwx 89 mapr mapr 87 Jul 30 00:04 ..
      drwxr-xr-x  2 mapr mapr  1 Jul 30 00:05 id2=20150101
      drwxr-xr-x  2 mapr mapr  1 Jul 30 00:05 id2=__HIVE_DEFAULT_PARTITION__
      drwxr-xr-x  4 mapr mapr  2 Jul 30 00:05 .
      

      3.Drill will fail:

      select * from h1_testpart2;
      Error: SYSTEM ERROR: NumberFormatException: For input string: "__HIVE_DEFAULT_PARTITION__"
      
      Fragment 0:0
      
      [Error Id: 509eb392-db9a-42f3-96ea-fb597425f49f on h1.poc.com:31010]
      
        (java.lang.reflect.UndeclaredThrowableException) null
          org.apache.hadoop.security.UserGroupInformation.doAs():1581
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
          org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
          org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
          org.apache.drill.common.SelfCleaningRunnable.run():38
          java.util.concurrent.ThreadPoolExecutor.runWorker():1142
          java.util.concurrent.ThreadPoolExecutor$Worker.run():617
          java.lang.Thread.run():745
        Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while initializing HiveRecordReader: For input string: "__HIVE_DEFAULT_PARTITION__"
          org.apache.drill.exec.store.hive.HiveRecordReader.init():241
          org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
          org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
          org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
          org.apache.drill.exec.physical.impl.ImplCreator$2.run():138
          org.apache.drill.exec.physical.impl.ImplCreator$2.run():136
          java.security.AccessController.doPrivileged():-2
          javax.security.auth.Subject.doAs():422
          org.apache.hadoop.security.UserGroupInformation.doAs():1566
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
          org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
          org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
          org.apache.drill.common.SelfCleaningRunnable.run():38
          java.util.concurrent.ThreadPoolExecutor.runWorker():1142
          java.util.concurrent.ThreadPoolExecutor$Worker.run():617
          java.lang.Thread.run():745
        Caused By (java.lang.NumberFormatException) For input string: "__HIVE_DEFAULT_PARTITION__"
          java.lang.NumberFormatException.forInputString():65
          java.lang.Integer.parseInt():580
          java.lang.Integer.parseInt():615
          org.apache.drill.exec.store.hive.HiveRecordReader.convertPartitionType():605
          org.apache.drill.exec.store.hive.HiveRecordReader.init():236
          org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
          org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
          org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
          org.apache.drill.exec.physical.impl.ImplCreator$2.run():138
          org.apache.drill.exec.physical.impl.ImplCreator$2.run():136
          java.security.AccessController.doPrivileged():-2
          javax.security.auth.Subject.doAs():422
          org.apache.hadoop.security.UserGroupInformation.doAs():1566
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():131
          org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
          org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
          org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
          org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
          org.apache.drill.common.SelfCleaningRunnable.run():38
          java.util.concurrent.ThreadPoolExecutor.runWorker():1142
          java.util.concurrent.ThreadPoolExecutor$Worker.run():617
          java.lang.Thread.run():745 (state=,code=0)
      

        Attachments

        1. DRILL-3579-1.patch
          64 kB
          Venki Korukanti

          Issue Links

            Activity

              People

              • Assignee:
                rkins Rahul Challapalli
                Reporter:
                haozhu Hao Zhu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: