Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      We've seen a situation where some work was failing from (recurrent) connection reset exceptions.

      Irrespective of the root cause, these were surfacing not in the read operations, but when the input stream was being closed -including during a seek()

      These exceptions could be caught & logged & warn, rather than trigger immediate failures. It shouldn't matter to the next GET whether the last stream closed prematurely, as long as the new one works

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Inner stack

          Caused by: java.net.SocketException: Connection reset
          	at java.net.SocketInputStream.read(SocketInputStream.java:196)
          	at java.net.SocketInputStream.read(SocketInputStream.java:122)
          	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
          	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
          	at sun.security.ssl.InputRecord.read(InputRecord.java:509)
          	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
          	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
          	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
          	at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:204)
          	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182)
          	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:204)
          	at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:108)
          	at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:164)
          	at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:237)
          	at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:186)
          	at org.apache.http.util.EntityUtils.consume(EntityUtils.java:87)
          	at org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream.releaseConnection(HttpMethodReleaseInputStream.java:102)
          	at org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream.close(HttpMethodReleaseInputStream.java:194)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.closeInnerStream(NativeS3FileSystem.java:176)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.updateInnerStream(NativeS3FileSystem.java:192)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.seek(NativeS3FileSystem.java:207)
          	at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:96)
          	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - Inner stack Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) at sun.security.ssl.InputRecord.read(InputRecord.java:509) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:204) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:204) at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:108) at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:164) at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:237) at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:186) at org.apache.http.util.EntityUtils.consume(EntityUtils.java:87) at org.jets3t.service.impl. rest .httpclient.HttpMethodReleaseInputStream.releaseConnection(HttpMethodReleaseInputStream.java:102) at org.jets3t.service.impl. rest .httpclient.HttpMethodReleaseInputStream.close(HttpMethodReleaseInputStream.java:194) at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.closeInnerStream(NativeS3FileSystem.java:176) at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.updateInnerStream(NativeS3FileSystem.java:192) at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.seek(NativeS3FileSystem.java:207) at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:96) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
          Hide
          tsato Takenori Sato added a comment -

          Isn't this the duplicate of HADOOP-11730?

          Show
          tsato Takenori Sato added a comment - Isn't this the duplicate of HADOOP-11730 ?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it's similar to HADOOP-11730; that's the overall "recover from failure" code. This is for close() to not trigger problems.

          HADOOP-11730 is probably a superset.

          Show
          stevel@apache.org Steve Loughran added a comment - it's similar to HADOOP-11730 ; that's the overall "recover from failure" code. This is for close() to not trigger problems. HADOOP-11730 is probably a superset.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This turns out to be a different symptom of the HADOOP-11570 problem; the chunked stream reader is trying to read to the end of the input stream.

          That patched S3a.close() to shut the stream down more aggressively. We don't have a patch for s3n to do the same. Looking at HADOOP-11570 though, it's vulnerable to the same problem of a clean close() triggering an exception. It needs a more robust close() operator too

          Show
          stevel@apache.org Steve Loughran added a comment - This turns out to be a different symptom of the HADOOP-11570 problem; the chunked stream reader is trying to read to the end of the input stream. That patched S3a.close() to shut the stream down more aggressively. We don't have a patch for s3n to do the same. Looking at HADOOP-11570 though, it's vulnerable to the same problem of a clean close() triggering an exception. It needs a more robust close() operator too

            People

            • Assignee:
              tsato Takenori Sato
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development