Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-783

H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.10.0, 1.8.2
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      ParquetFileReader opens a SeekableInputStream to read a footer. In the process, it opens a new FSDataInputStream and wraps it. However, H2SeekableInputStream does not override the close method. Therefore, when ParquetFileReader closes it, the underlying FSDataInputStream is not closed. As a result, these stale connections can exhaust a clusters' data nodes' connection resources and lead to mysterious HDFS read failures in HDFS clients, e.g.

      org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-905337612-172.16.70.103-1444328960665:blk_1720536852_646811517
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                michael Michael Allman
                Reporter:
                michael Michael Allman
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: