[HDFS-16068] WebHdfsFileSystem has a possible connection leak in connection with HttpFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 2.10.2, 3.2.3, 3.3.2
• Fix Version/s: 3.4.0, 2.10.2, 3.2.3, 3.3.2
• Component/s: webhdfs
• Labels: pull-request-available
• Target Version/s: 3.4.0, 2.10.2, 3.2.3, 3.3.2
• Hadoop Flags: Reviewed
• Assignee: Takanobu Asanuma
• Reporter: Takanobu Asanuma
• Created: 15/Jun/21 01:18
• Updated: 11/Feb/24 14:18
• Resolved: 15/Jun/21 07:50

Description

When we use WebHdfsFileSystem for HttpFS, some connections remain for a while after the filesystems are closed until GC runs. After investigating it for a while, I found that there is a potential connection leak in WebHdfsFileSystem.

// Close both the InputStream and the connection.
@VisibleForTesting
void closeInputStream(RunnerState rs) throws IOException {
  if (in != null) {
    IOUtils.close(cachedConnection);
    in = null;
  }
  cachedConnection = null;
  runnerState = rs;
}

In the above code, if the variable of in is null and cachedConnection is not null, cachedConnection doesn't close and the connection remains. I think this is the cause of our problem.

Issue Links

• links to: GitHub Pull Request #3104