Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-14070

S3a: Failed to reset the request input stream/make S3A uploadPart() retriable

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Feb 07, 2017 8:05:46 AM com.google.common.util.concurrent.Futures$CombinedFuture setExceptionAndMaybeLog
      SEVERE: input future failed.
      com.amazonaws.ResetException: Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1221)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1042)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4041)
      at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3041)
      at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3026)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:1114)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:501)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:492)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1219)
      at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Resetting to invalid mark
      at java.io.BufferedInputStream.reset(BufferedInputStream.java:448)
      at com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:169)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
      ... 20 more
      2017-02-07 08:05:46 WARN S3AInstrumentation:777 - Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics{blocksSubmitted=519, blocksInQueue=0, blocksActive=1, blockUploadsCompleted=518, blockUploadsFailed=2, bytesPendingUpload=82528300, bytesUploaded=54316236800, blocksAllocated=519, blocksReleased=519, blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=2637812 ms, queueDuration=839 ms, averageQueueTime=1 ms, totalUploadDuration=2638651 ms, effectiveBandwidth=2.05848506680118E7 bytes/s}
      Exception in thread "main" org.apache.hadoop.fs.s3a.AWSClientIOException: Multi-part upload with id 'uDonLgtsyeToSmhyZuNb7YrubCDiyXCCQy4mdVc5ZmYWPPHyZ3H3ZlFZzKktaPUiYb7uT4.oM.lcyoazHF7W8pK4xWmXV4RWmIYGYYhN6m25nWRrBEE9DcJHcgIhFD8xd7EKIjijEd1k4S5JY1HQvA--' to 2017/history-170130.orc on 2017/history-170130.orc: com.amazonaws.ResetException: Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int): Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
      at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128)
      at org.apache.hadoop.fs.s3a.S3AUtils.extractException(S3AUtils.java:200)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.waitForAllPartUploads(S3ABlockOutputStream.java:539)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$100(S3ABlockOutputStream.java:456)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:351)
      at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
      at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
      at org.apache.orc.impl.PhysicalFsWriter.close(PhysicalFsWriter.java:221)
      at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2827)
      at net.mojodna.osm2orc.standalone.OsmPbf2Orc.convert(OsmPbf2Orc.java:296)
      at net.mojodna.osm2orc.Osm2Orc.main(Osm2Orc.java:47)
      Caused by: com.amazonaws.ResetException: Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1221)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1042)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4041)
      at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3041)
      at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3026)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:1114)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:501)
      at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:492)
      at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
      at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Resetting to invalid mark
      at java.io.BufferedInputStream.reset(BufferedInputStream.java:448)
      at com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:169)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1219)
      ... 20 more
      

      Potentially relevant: https://github.com/aws/aws-sdk-java/issues/427

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mojodna Seth Fitzsimmons
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: