[HIVE-8870] errors when selecting a struct field within an array from ORC based tables

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0, 0.14.0
• Fix Version/s: 1.0.0
• Component/s: File Formats, Query Processor

Environment

HDP 2.1 / HDP 2.2 (YARN, but no Tez)

• Assignee: Sergio Peña Assign to me
• Reporter: Michael Haeusler
• Created: 14/Nov/14 13:27
• Updated: 19/Feb/15 18:22
• Resolved: 05/Dec/14 08:46

Description

When using ORC as storage for a table, we get errors on selecting a struct field within an array. These errors do not appear with default format.

CREATE  TABLE `foobar_orc`(
  `uid` bigint,
  `elements` array<struct<elementid:bigint,foo:struct<bar:string>>>)
STORED AS ORC;

When selecting from this empty table, we get a direct NPE within the Hive CLI:

SELECT
  elements.elementId
FROM
  foobar_orc;
-- FAILED: RuntimeException java.lang.NullPointerException

A more real-world query produces a RuntimeException / NullPointerException in the mapper:

SELECT
  uid,
  element.elementId
FROM
  foobar_orc
LATERAL VIEW
  EXPLODE(elements) e AS element;


Error: java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
[...]
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:61)
[...]
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

Both queries run fine on a non-orc table:

CREATE  TABLE `foobar`(
  `uid` bigint,
  `elements` array<struct<elementid:bigint,foo:struct<bar:string>>>);  

SELECT
  elements.elementId
FROM
  foobar;
-- OK
-- Time taken: 0.225 seconds

SELECT
  uid,
  element.elementId
FROM
  foobar
LATERAL VIEW
  EXPLODE(elements) e AS element;
-- Total MapReduce CPU Time Spent: 1 seconds 920 msec
-- OK
-- Time taken: 25.905 seconds

Issue Links

• duplicates: HIVE-6198 ORC file and struct column names are case sensitive
• is duplicated by: HIVE-9107 Non-lowercase field names in structs causes NullPointerException