Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16068

WebHdfsFileSystem has a possible connection leak in connection with HttpFS

    XMLWordPrintableJSON

    Details

      Description

      When we use WebHdfsFileSystem for HttpFS, some connections remain for a while after the filesystems are closed until GC runs. After investigating it for a while, I found that there is a potential connection leak in WebHdfsFileSystem.

      // Close both the InputStream and the connection.
      @VisibleForTesting
      void closeInputStream(RunnerState rs) throws IOException {
        if (in != null) {
          IOUtils.close(cachedConnection);
          in = null;
        }
        cachedConnection = null;
        runnerState = rs;
      }
      

      In the above code, if the variable of in is null and cachedConnection is not null, cachedConnection doesn't close and the connection remains. I think this is the cause of our problem.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tasanuma Takanobu Asanuma
                Reporter:
                tasanuma Takanobu Asanuma
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m