Apache Drill
  1. Apache Drill
  2. DRILL-960

Doing a select on a orc partitioned hive table fails with ClassCastException

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Storage - Hive
    • Labels:
      None

      Description

      SELECT id FROM orc_create_people;

      Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "dcfd06bf-9239-44cf-8f16-f92ffac39005"
      endpoint

      { address: "drillats4.qa.lab" user_port: 31010 control_port: 31011 data_port: 31012 }

      error_type: 0
      message: "Failure while running fragment. < ClassCastException:[ org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector ]"
      ]
      Error: exception while executing query (state=,code=0)

      1. orc_complex.q
        0.6 kB
        Ramana Inukonda Nagaraj
      2. orc_create_people.txt
        4 kB
        Ramana Inukonda Nagaraj
      3. orc_physicalplan.json
        5 kB
        Ramana Inukonda Nagaraj

        Activity

        Ramana Inukonda Nagaraj created issue -
        Hide
        Ramana Inukonda Nagaraj added a comment -
        2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment.
        java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector
                at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
                at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
        2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile {
          state: FAILED
        
        Show
        Ramana Inukonda Nagaraj added a comment - 2014-06-11 21:19:17,526 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error dcfd06bf-9239-44cf-8f16-f92ffac39005: Failure while running fragment. java.lang.ClassCastException: org.apache.drill.exec.vector.Var16CharVector cannot be cast to org.apache.drill.exec.vector.VarCharVector at org.apache.drill.exec.store.hive.HiveRecordReader.populatePartitionVectors(HiveRecordReader.java:307) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:497) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:131) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:68) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:45) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:84) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang. Thread .run( Thread .java:744) [na:1.7.0_45] 2014-06-11 21:19:17,529 [20e3c074-a67e-4fa1-9852-8e24bf574cea:frag:1:0] DEBUG o.a.d.e.w.f.NonRootStatusReporter - Sending status change message message to remote node: profile { state: FAILED
        Hide
        Ramana Inukonda Nagaraj added a comment -

        PFA the hive create table and the data file.
        git.commit.id.abbrev=27a9c98

        Show
        Ramana Inukonda Nagaraj added a comment - PFA the hive create table and the data file. git.commit.id.abbrev=27a9c98
        Hide
        Ramana Inukonda Nagaraj added a comment -

        orc create table script

        Show
        Ramana Inukonda Nagaraj added a comment - orc create table script
        Ramana Inukonda Nagaraj made changes -
        Field Original Value New Value
        Attachment orc_complex.q [ 12649885 ]
        Hide
        Ramana Inukonda Nagaraj added a comment -

        orc data file.

        Show
        Ramana Inukonda Nagaraj added a comment - orc data file.
        Ramana Inukonda Nagaraj made changes -
        Attachment orc_create_people.txt [ 12649886 ]
        Hide
        Steven Phillips added a comment -

        I don't think this problem is specific to orc files. There are two problems I see here:

        We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException

        The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it.

        Can you upload the physical plan for this query a an attachment?

        Show
        Steven Phillips added a comment - I don't think this problem is specific to orc files. There are two problems I see here: We have some code (Types.getMajorTypeFromName()) which converts "string" to VARCHAR16. But elsewhere in hive reader we treat "string" as VARCHAR. So that explains the ClassCastException The other issue is the fact that we are trying to populate a partition vector at all. Since this query is not selecting a partition column, we shouldn't be populating it. Can you upload the physical plan for this query a an attachment?
        Hide
        Ramana Inukonda Nagaraj added a comment -

        Physical plan

        Show
        Ramana Inukonda Nagaraj added a comment - Physical plan
        Ramana Inukonda Nagaraj made changes -
        Attachment orc_physicalplan.json [ 12649914 ]
        Steven Phillips made changes -
        Assignee Venki Korukanti [ vkorukanti ] Steven Phillips [ sphillips ]
        Hide
        Steven Phillips added a comment -

        Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns.

        As for the ClassCastException, this looks to be due to DRILL-452, which patch has the code that treats "string" as VARCHAR16.

        Show
        Steven Phillips added a comment - Okay, so it looks like Project Pushdown is not enable for the Hive reader. So we are materializing all of the columns. As for the ClassCastException, this looks to be due to DRILL-452 , which patch has the code that treats "string" as VARCHAR16.
        Steven Phillips made changes -
        Assignee Steven Phillips [ sphillips ] Venki Korukanti [ vkorukanti ]
        Hide
        Venki Korukanti added a comment -

        Fixed as part of DRILL-735.

        Show
        Venki Korukanti added a comment - Fixed as part of DRILL-735 .
        Venki Korukanti made changes -
        Assignee Venki Korukanti [ vkorukanti ] Jacques Nadeau [ jnadeau ]
        Jacques Nadeau made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ramana Inukonda Nagaraj made changes -
        Assignee Jacques Nadeau [ jnadeau ] Ramana Inukonda Nagaraj [ inramana ]
        Jacques Nadeau made changes -
        Fix Version/s 0.4.0 [ 12324963 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail, testing [ 12868818 ] Drill workflow [ 12933358 ]

          People

          • Assignee:
            Ramana Inukonda Nagaraj
            Reporter:
            Ramana Inukonda Nagaraj
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development