Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11907

SqsIO checkpoint causes throttling

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: 2.27.0
    • Fix Version/s: 2.32.0
    • Component/s: io-java-aws
    • Labels:
      None

      Description

      SqsIO currently calls the delete message API for each message when finalizing checkpoints. 

      This creates large amount of requests to the AWS API, which are frequently throttled and t throw the exception:

      Exception in thread "main" com.amazonaws.services.sqs.model.AmazonSQSException: Request is throttled. (Service: AmazonSQS; Status Code: 403; Error Code: RequestThrottled; Request ID: XXXXXXXXXXXXX) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2207) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2174) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2163) at com.amazonaws.services.sqs.AmazonSQSClient.executeDeleteMessage(AmazonSQSClient.java:893) at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessage(AmazonSQSClient.java:865) at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessage(AmazonSQSClient.java:905) at org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.delete(SqsUnboundedReader.java:131) at org.apache.beam.sdk.io.aws.sqs.SqsCheckpointMark.lambda$finalizeCheckpoint$0(SqsCheckpointMark.java:43) at java.util.Optional.ifPresent(Optional.java:159) at org.apache.beam.sdk.io.aws.sqs.SqsCheckpointMark.finalizeCheckpoint(SqsCheckpointMark.java:43) at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.finishRead(UnboundedReadEvaluatorFactory.java:254) at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:147) at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160) at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dennisyung Dennis Yung
                Reporter:
                dennisyung Dennis Yung
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 9h 40m
                  9h 40m