[HADOOP-18353] HEAD OBJECT returns only 400 BAD REQUEST when token is expired

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 3.3.3
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Mukund Thakur
• Created: 20/Jul/22 17:42
• Updated: 24/Aug/22 08:16
• Resolved: 05/Aug/22 16:04

Description

I tried reproducing this today by changing this test https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116 . Getting a session token for 15 mins and trying every one mins and finally it fails after 15 mins. 
 
Looks like the AWS SDK is not having the Expired Token error message as I could see the same in access logs but I see BadRequest on the SDK logs. 
 
S3A Connector logs with SDK debug enabled.

2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from TemporaryAWSCredentialsProvider
2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD https://mthakur-us-west-1.s3.us-west-1.amazonaws.com /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers: (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type: application/octet-stream, Referer: https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318, User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy, )
2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1846)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: WMGQ0KC4MHEMZTQC; S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=; Proxy: null), S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is closed

AWS access logs

183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 - WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" 400 ExpiredToken 556 - 5 - "https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy" - IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q= SigV4 ECDHE-RSA-AES128-SHA AuthHeader mthakur-us-west-1.s3.us-west-1.amazonaws.com TLSv1.2 -
 
I tested by running repeatedly ITestCustomSigner in S3A, and also just ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and I can reproduce by running this after credential expiry.

 

aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key test-object –debug

 
To summarise:

• If I run ListObjectsV2, I get “400 ExpiredToken”{}{}
• If I run HeadObject, I get “400 Bad Request”{}{}
• If I run GetObject, I get “400 ExpiredToken”