Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4160 Vectorized Query Execution in Hive
  3. HIVE-4665

error at VectorExecMapper.close in group-by-agg query over ORC, vectorized

    XMLWordPrintableJSON

    Details

      Description

      CREATE EXTERNAL TABLE FactSqlEngineAM4712( dAppVersionBuild int, dAppVersionBuildUNMAPPED32449 int, dAppVersionMajor int, dAppVersionMinor32447 int, dAverageCols23083 int, dDatabaseSize23090 int, dDate string, dIsInternalMSFT16431 int, dLockEscalationDisabled23323 int, dLockEscalationEnabled23324 int, dMachineID int, dNumberTables23008 int, dNumCompressionPagePartitions23088 int, dNumCompressionRowPartitions23089 int, dNumIndexFragmentation23084 int, dNumPartitionedTables23098 int, dNumPartitions23099 int, dNumTablesClusterIndex23010 int, dNumTablesHeap23100 int, dSessionType5618 int, dSqlEdition8213 int, dTempDbSize23103 int, mNumColumnStoreIndexesVar48171 bigint, mOccurrences int, mRowFlag int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/user/ehans/SQM';

      create table FactSqlEngineAM_vec_ORC ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' stored as INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.CommonOrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' AS select * from FactSqlEngineAM4712;

      hive> select ddate, max(dnumbertables23008) from factsqlengineam_vec_orc group by ddate;

      Total MapReduce jobs = 1
      Launching Job 1 out of 1
      Number of reduce tasks not specified. Estimated from input data size: 3
      In order to change the average load for a reducer (in bytes):
      set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
      set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
      set mapred.reduce.tasks=<number>
      Validating if vectorized execution is applicable
      Going down the vectorization path
      java.lang.InstantiationException: org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector
      Continuing ...
      java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();
      Continuing ...
      java.lang.InstantiationException: org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator
      Continuing ...
      java.lang.Exception: XMLEncoder: discarding statement ArrayList.add(VectorGroupByOperator);
      Continuing ...
      Starting Job = job_201306041757_0016, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201306041757_0016
      Kill Command = c:\Hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd job -kill job_201306041757_0016
      Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 3
      2013-06-05 10:03:06,022 Stage-1 map = 0%, reduce = 0%
      2013-06-05 10:03:51,142 Stage-1 map = 100%, reduce = 100%
      Ended Job = job_201306041757_0016 with errors
      Error during job, obtaining debugging information...
      Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201306041757_0016
      Examining task ID: task_201306041757_0016_m_000009 (and more) from job job_201306041757_0016

      Task with the most failures(4):


      Task ID:
      task_201306041757_0016_m_000000

      URL:
      http://localhost:50030/taskdetails.jsp?jobid=job_201306041757_0016&tipid=task_201306041757_0016_m_000000


      Diagnostic Messages for this Task:
      java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:229)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
      at org.apache.hadoop.mapred.Child.main(Child.java:265)
      Caused by: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be cast to org.apache.hadoop.io.Text
      at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(Writable
      StringObjectInspector.java:40)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:481)
      at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:235)
      at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
      at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:253)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
      at org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:196)
      ... 8 more

      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
      MapReduce Jobs Launched:
      Job 0: Map: 8 Reduce: 3 HDFS Read: 0 HDFS Write: 0 FAIL
      Total MapReduce CPU Time Spent: 0 msec
      hive>

        Attachments

        1. HIVE-4665.2.patch
          48 kB
          Jitendra Nath Pandey
        2. HIVE-4665.1.patch
          47 kB
          Jitendra Nath Pandey

          Activity

            People

            • Assignee:
              jnp Jitendra Nath Pandey
              Reporter:
              ehans Eric N. Hanson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: