Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-19210

s3a: TestS3AAWSCredentialsProvider and TestS3AInputStreamRetry really slow

    XMLWordPrintableJSON

Details

    Description

      Not noticed this before, but the unit tests TestS3AAWSCredentialsProvider and TestS3AInputStreamRetry are so slow they will be hurting over all test performance times: no integration tests will start until these are all complete.

      
      mvn test -T 1C -Dparallel-tests
      
      ...
      [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.877 s - in org.apache.hadoop.fs.s3a.TestS3AInputStreamRetry
      ...
      [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.038 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
      

      The PR cuts total execution time of a 10 thread test run from 3 minutes to 2:30

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: