Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3893

Issue with Drill after Hive Alters the Table

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.1.0
    • Fix Version/s: 1.4.0
    • Labels:
      None
    • Environment:

      DEV

      Description

      I reproduced this again on another partitioned table with existing data.

      Providing some more details. I have enabled the version mode for errors. Drill is unable to fetch the new column name that was introduced.This most likely to me seems to me that it’s still picking up the stale metadata of hive.

      if (!tableColumns.contains(columnName)) {
      if (partitionNames.contains(columnName))

      { selectedPartitionNames.add(columnName); }

      else

      { throw new ExecutionSetupException(String.format("Column %s does not exist", columnName)); }

      }

      select testdata from testtable;
      Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exist

      Fragment 0:0

      [Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on x.x.com:9999]

      (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while initializing HiveRecordReader: Column testdata does not exist
      org.apache.drill.exec.store.hive.HiveRecordReader.init():241
      org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
      org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
      org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
      org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
      org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
      org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
      org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
      org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
      org.apache.drill.common.SelfCleaningRunnable.run():38
      java.util.concurrent.ThreadPoolExecutor.runWorker():1142
      java.util.concurrent.ThreadPoolExecutor$Worker.run():617
      java.lang.Thread.run():745
      Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Column testdata does not exist
      org.apache.drill.exec.store.hive.HiveRecordReader.init():206
      org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
      org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
      org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
      org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
      org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
      org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
      org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
      org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
      org.apache.drill.common.SelfCleaningRunnable.run():38
      java.util.concurrent.ThreadPoolExecutor.runWorker():1142
      java.util.concurrent.ThreadPoolExecutor$Worker.run():617
      java.lang.Thread.run():745 (state=,code=0)

      #########################################################

      Please note that this is a partitioned table with existing data.

      Does Drill Cache the Meta somewhere and hence it’s not getting reflected immediately ?

      DRILL CLI
      > select x from xx;
      Error: SYSTEM ERROR: ExecutionSetupException: Column x does not exist

      Fragment 0:0

      [Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on x.x.com:999] (state=,code=0)

      HIVE CLI
      hive> describe formatted x;
      OK

      1. col_name data_type comment

      <columns>

        Attachments

          Activity

            People

            • Assignee:
              vkorukanti Venki Korukanti
              Reporter:
              arnab1980 arnab chatterjee
              Reviewer:
              Krystal
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: