[SPARK-23399] Register a task completion listener first for OrcColumnarBatchReader

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 12/Feb/18 21:45
• Updated: 23/Feb/18 06:34
• Resolved: 14/Feb/18 02:57

Description

This is related with SPARK-23390.

Currently, there was a opened file leak for OrcColumnarBatchReader.

[info] - Enabling/disabling ignoreMissingFiles using orc (648 milliseconds)
15:55:58.673 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 61.0 (TID 85, localhost, executor driver): TaskKilled (Stage cancelled)
15:55:58.674 WARN org.apache.spark.DebugFilesystem: Leaked filesystem connection created at:
java.lang.Throwable
	at org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:36)
	at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:70)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
	at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.open(RecordReaderUtils.java:173)
	at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:254)
	at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:633)
	at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:138)

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• relates to: SPARK-23390 Flaky test: FileBasedDataSourceSuite
• links to: [Github] Pull Request #20590 (dongjoon-hyun)