Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-960

Doing a select on a orc partitioned hive table fails with ClassCastException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      SELECT id FROM orc_create_people;

      Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "dcfd06bf-9239-44cf-8f16-f92ffac39005"
      endpoint {
      address: "drillats4.qa.lab"
      user_port: 31010
      control_port: 31011
      data_port: 31012
      }
      error_type: 0
      message: "Failure while running fragment. < ClassCastException:[ org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector ]"
      ]
      Error: exception while executing query (state=,code=0)

      1. orc_complex.q
        0.6 kB
        Ramana Inukonda Nagaraj
      2. orc_create_people.txt
        4 kB
        Ramana Inukonda Nagaraj
      3. orc_physicalplan.json
        5 kB
        Ramana Inukonda Nagaraj

        Activity

        Hide
        vkorukanti Venki Korukanti added a comment -

        Fixed as part of DRILL-735.

        Show
        vkorukanti Venki Korukanti added a comment - Fixed as part of DRILL-735 .
        Hide
        sphillips Steven Phillips added a comment -

        Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns.

        As for the ClassCastException, this looks to be due to DRILL-452, which patch has the code that treats "string" as VARCHAR16.

        Show
        sphillips Steven Phillips added a comment - Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns. As for the ClassCastException, this looks to be due to DRILL-452 , which patch has the code that treats "string" as VARCHAR16.
        Hide
        inramana Ramana Inukonda Nagaraj added a comment -

        Physical plan

        Show
        inramana Ramana Inukonda Nagaraj added a comment - Physical plan
        Hide
        sphillips Steven Phillips added a comment -

        I don't think this problem is specific to orc files. There are two problems I see here:

        We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException

        The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it.

        Can you upload the physical plan for this query a an attachment?

        Show
        sphillips Steven Phillips added a comment - I don't think this problem is specific to orc files. There are two problems I see here: We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it. Can you upload the physical plan for this query a an attachment?
        Hide
        inramana Ramana Inukonda Nagaraj added a comment -

        orc data file.

        Show
        inramana Ramana Inukonda Nagaraj added a comment - orc data file.
        Hide
        inramana Ramana Inukonda Nagaraj added a comment -

        orc create table script

        Show
        inramana Ramana Inukonda Nagaraj added a comment - orc create table script
        Hide
        inramana Ramana Inukonda Nagaraj added a comment -

        PFA the hive create table and the data file.
        git.commit.id.abbrev=27a9c98

        Show
        inramana Ramana Inukonda Nagaraj added a comment - PFA the hive create table and the data file. git.commit.id.abbrev=27a9c98
        Hide
        inramana Ramana Inukonda Nagaraj added a comment -
        2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment.
        java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector
                at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
                at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
        2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile {
          state: FAILED
        
        Show
        inramana Ramana Inukonda Nagaraj added a comment - 2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment. java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang. Thread .run( Thread .java:744) [na:1.7.0_45] 2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile { state: FAILED

          People

          • Assignee:
            inramana Ramana Inukonda Nagaraj
            Reporter:
            inramana Ramana Inukonda Nagaraj
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development