[HADOOP-18201] Remove base and bucket overrides for endpoint in ITestS3ARequesterPays.java

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.3
• Fix Version/s: 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Daniel Carl Jones
• Reporter: Mehakmeet Singh
• Created: 13/Apr/22 13:42
• Updated: 14/Apr/22 16:02
• Resolved: 14/Apr/22 16:02

Description

If a user has set the endpoint for their bucket in a test environment, we should ignore that in a test that won't use the bucket that the endpoint is set for. In this case, we are using "s3a://usgs-landsat/" which is in the region us-west-2, and would fail if the user has explicitly set the endpoint to something else.

Example (I have set the endpoint to ap-south-1):

[ERROR] testRequesterPaysDisabledFails(org.apache.hadoop.fs.s3a.ITestS3ARequesterPays)  Time elapsed: 9.323 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSRedirectException: getFileStatus on s3a://usgs-landsat/collection02/catalog.json: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-west-2. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: Z09V8PMEEN5PHDRZ; S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=; Proxy: null), S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=:301 Moved Permanently: The bucket is in this region: us-west-2. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: Z09V8PMEEN5PHDRZ; S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=; Proxy: null)
 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:233)
 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3440)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3346)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:4890)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$executeOpen$6(S3AFileSystem.java:1437)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.executeOpen(S3AFileSystem.java:1435)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1409)
  
<......>

CC: [e-mail]  mthakur  dannycjones

Issue Links

• is caused by: HADOOP-14661 S3A to support Requester Pays Buckets
• is related to: HADOOP-18194 Public dataset class for S3A integration tests
• links to: GitHub Pull Request #4169