[ORC-525] Users must close Readers in ORC 1.5.6

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.6
• Component/s: Java
• Assignee: Owen O'Malley
• Reporter: Dongjoon Hyun
• Created: 27/Jun/19 07:14
• Updated: 29/Jun/19 04:10
• Resolved: 29/Jun/19 00:43

Description

Unlike 1.5.5, ORC 1.5.6-rc1 seems to have a following issue which causes Apache Spark unit test failures.

$ build/sbt "sql/testOnly *.SQLQuerySuite"
...
[info]   Cause: java.lang.Throwable:
[info]   at org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:36)
[info]   at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:70)
[info]   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
[info]   at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:535)
[info]   at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:368)
[info]   at org.apache.orc.OrcFile.createReader(OrcFile.java:343)

ORC-498 (ReaderImpl and RecordReaderImpl open separate file handles) seems to be the reason of this change.

Issue Links

• is caused by: SPARK-28208 Upgrade to ORC 1.5.6