[PARQUET-783] H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0, 1.8.2
• Component/s: parquet-mr
• Assignee: Michael MacFadden
• Reporter: Michael MacFadden
• Created: 28/Nov/16 20:37
• Updated: 21/Apr/18 12:39
• Resolved: 05/Dec/16 23:29

Description

ParquetFileReader opens a SeekableInputStream to read a footer. In the process, it opens a new FSDataInputStream and wraps it. However, H2SeekableInputStream does not override the close method. Therefore, when ParquetFileReader closes it, the underlying FSDataInputStream is not closed. As a result, these stale connections can exhaust a clusters' data nodes' connection resources and lead to mysterious HDFS read failures in HDFS clients, e.g.

org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-905337612-172.16.70.103-1444328960665:blk_1720536852_646811517

Issue Links

• blocks: PARQUET-1027 release Parquet-mr 1.9.1
• links to: PR