Apache Drill
  1. Apache Drill
  2. DRILL-960

Doing a select on a orc partitioned hive table fails with ClassCastException

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      SELECT id FROM orc_create_people;

      Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "dcfd06bf-9239-44cf-8f16-f92ffac39005"
      endpoint

      { address: "drillats4.qa.lab" user_port: 31010 control_port: 31011 data_port: 31012 }

      error_type: 0
      message: "Failure while running fragment. < ClassCastException:[ org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector ]"
      ]
      Error: exception while executing query (state=,code=0)

      1. orc_complex.q
        0.6 kB
        Ramana Inukonda Nagaraj
      2. orc_create_people.txt
        4 kB
        Ramana Inukonda Nagaraj
      3. orc_physicalplan.json
        5 kB
        Ramana Inukonda Nagaraj

        Activity

        Hide
        Venki Korukanti added a comment -

        Fixed as part of DRILL-735.

        Show
        Venki Korukanti added a comment - Fixed as part of DRILL-735 .
        Hide
        Steven Phillips added a comment -

        Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns.

        As for the ClassCastException, this looks to be due to DRILL-452, which patch has the code that treats "string" as VARCHAR16.

        Show
        Steven Phillips added a comment - Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns. As for the ClassCastException, this looks to be due to DRILL-452 , which patch has the code that treats "string" as VARCHAR16.
        Hide
        Ramana Inukonda Nagaraj added a comment -

        Physical plan

        Show
        Ramana Inukonda Nagaraj added a comment - Physical plan
        Hide
        Steven Phillips added a comment -

        I don't think this problem is specific to orc files. There are two problems I see here:

        We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException

        The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it.

        Can you upload the physical plan for this query a an attachment?

        Show
        Steven Phillips added a comment - I don't think this problem is specific to orc files. There are two problems I see here: We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it. Can you upload the physical plan for this query a an attachment?
        Hide
        Ramana Inukonda Nagaraj added a comment -

        orc data file.

        Show
        Ramana Inukonda Nagaraj added a comment - orc data file.
        Hide
        Ramana Inukonda Nagaraj added a comment -

        orc create table script

        Show
        Ramana Inukonda Nagaraj added a comment - orc create table script
        Hide
        Ramana Inukonda Nagaraj added a comment -

        PFA the hive create table and the data file.
        git.commit.id.abbrev=27a9c98

        Show
        Ramana Inukonda Nagaraj added a comment - PFA the hive create table and the data file. git.commit.id.abbrev=27a9c98
        Hide
        Ramana Inukonda Nagaraj added a comment -
        2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment.
        java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector
                at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
                at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
        2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile {
          state: FAILED
        
        Show
        Ramana Inukonda Nagaraj added a comment - 2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment. java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang. Thread .run( Thread .java:744) [na:1.7.0_45] 2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile { state: FAILED

          People

          • Assignee:
            Ramana Inukonda Nagaraj
            Reporter:
            Ramana Inukonda Nagaraj
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development