Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10203

Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8)

    • Hadoop Flags:
      Reviewed

      Description

      Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.

      This affects any client that tries to read many small files very quickly (e.g. distcp from s3 to hdfs with small files blocks due to connection pool starvation).

      This is not a problem for larger files because when the GC runs any connection that's out of scope will be released in #finalize().

      We are seeing the following log messages as a symptom of this problem:

      13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation.
      13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully released HttpMethod in finalize(). You were lucky this time... Please ensure response data streams are always fully consumed or closed.
      

        Attachments

        1. HADOOP-10203-trunk.patch
          2 kB
          Andrei Savu
        2. HADOOP-10203.patch
          2 kB
          Andrei Savu

          Activity

            People

            • Assignee:
              savu.andrei Andrei Savu
              Reporter:
              savu.andrei Andrei Savu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: