[HIVE-17940] IllegalArgumentException when reading last row-group in an ORC stripe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.2, 1.3.0
• Fix Version/s: 1.2.3, 1.3.0
• Component/s: ORC
• Assignee: Chris Drome
• Reporter: Mithun Radhakrishnan
• Created: 30/Oct/17 20:50
• Updated: 01/Dec/17 19:30
• Resolved: 01/Dec/17 19:30

Description

(This is a backport of HIVE-10024 to branch-1.2, and branch-1.)

When the last row-group in an ORC stripe contains fewer records than specified in ${orc.row.index.stride}, and if a column value is sparse (i.e. mostly nulls), then one sees the following failure when reading the ORC stripe:

 java.lang.IllegalArgumentException: Seek in Stream for column 82 kind DATA to 130 is outside of the data
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: Seek in Stream for column 82 kind DATA to 130 is outside of the data
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:71)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:322)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
... 14 more

sershe had a fix for this in HIVE-10024, in branch-2. After running into this in production with branch-1+, we find that the fix for HIVE-10024 sorts this out in branch-1 as well.

This is a fairly rare case, but it leads to bad reads on valid ORC files. I will back-port this shortly.

Issue Links

• is blocked by: HIVE-17949 itests compile is busted on branch-1.2