[HDDS-4856] Ruby S3 SDK never get authenticated by Ozone

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Component/s: S3
• Labels: pull-request-available

Environment

Secure setup of Ozone 1.0.0

• Target Version/s: 1.2.0
• Assignee: Marton Elek
• Reporter: UENISHI Kota
• Created: 22/Feb/21 12:49
• Updated: 23/Mar/21 14:34
• Resolved: 23/Mar/21 14:34

Description

When the very first call by Ruby client against secure setup of Ozone, the server returns 400 no matter how valid the request is. See the attached ruby-sdk-patch.diff, which adds some tests on S3 auth header signature-to-sign generation. It consists of two test additions, the "2" is the one generated by boto3, the "3" is generated by aws-ruby-sdk. Both passes the additional tests, which are definitely valid.

However, when real HTTP request is sent by Ruby client, e.g. ozone-test.rb attached, it fails with 400. The header was like this (though the host names and domains are masked):

GET //ozone.example.com:9879/sandbox?list-type=2&max-keys=1 HTTP/1.1
Content-Type:
Accept-Encoding:
User-Agent: aws-sdk-ruby3/3.112.0 ruby/2.7.2 x86_64-linux aws-sdk-s3/1.88.1
Host: ozone.example.com:9879
X-Amz-Date: 20210222T110554Z
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=[e-mail]/20210222/foobar/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0c9469f018f5
b3fd2cff6f8d4e4963f50aa71c6704def59527634404f5fc98a9
Content-Length: 0
Accept: /

On the other hand, request headers made by boto3 was:

GET //ozone.example.com:9879/sandbox?list-type=2&encoding-type=url HTTP/1.1
Host: ozone.example.com:9879
Accept-Encoding: identity
User-Agent: Boto3/1.17.12 Python/3.9.1 Linux/5.10.14-arch1-1 Botocore/1.20.12
X-Amz-Date: 20210222T110829Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=[e-mail]/20210222/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=94302f21cccac8832d3e
4fe25c5f6d8a0307188fb0e1b1983264339381d21dac

The difference of these requests are IMHO, "Content-Type" and "Accept-Encoding" are both empty in Ruby SDK. I'm afraid this error stems from partly Ruby SDK and partly from Jetty Issue. The former sends empty header lines and the latter rejects them.

And the s3g debug log (only error'ish part) follows:

2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: chain=NoCacheFilter@5e600dd5==org.apache.hadoop.hdds.server.http.NoCacheFilter,inst=true,async=true-
>safety@63a12c68==org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter,inst=true,async=true->info-page-redirect@576d5deb==org.apache.hadoop.ozone.s3.RootPageDis
playFilter,inst=true,async=false->jaxrs@603a422==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=1,inst=true,async=false
2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter NoCacheFilter@5e600dd5==org.apache.hadoop.hdds.server.http.NoCacheFilter,inst=true,async
=true
2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter safety@63a12c68==org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter,inst=
true,async=true
2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter info-page-redirect@576d5deb==org.apache.hadoop.ozone.s3.RootPageDisplayFilter,inst=true,
async=false
2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call servlet jaxrs@603a422==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=1,inst=true
,async=false
2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: sendError HttpChannelState@4893b376

Unknown macro: {s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=tru e al=0}

2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.session: Leaving scope org.eclipse.jetty.server.session.SessionHandler367746789==dftMaxIdleSec=-1 dispatch=REQUEST, a
sync=false, session=null, oldsession=null, oldsessionhandler=null
2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.Server: handled=true async=false committed=true on HttpChannelOverHttp@769bb34b{s=HttpChannelState@4893b376

Unknown macro: {s=HANDLIN G rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true al=0}

,r=1,c=false/false,a=HANDLING,uri=https://ozone.example.com:9879/sandbox?list-type=2&ma
x-keys=1,age=2}
2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: unhandle HttpChannelState@4893b376

Unknown macro: {s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true al=0}

2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: nextAction(false) SEND_ERROR HttpChannelState@4893b376

Unknown macro: {s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=f alse se=false i=false al=0}

Issue Links

• links to: GitHub Pull Request #2013