Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5271

S3G: Upload with Content-Type: plain/text; charset=utf-8 header will fail

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.1.0
    • None
    • Ozone Datanode, S3
    • None

    Description

      Upload content via S3 gateway with "Content-Type: plain/text; charset=utf-8" header will fail with 500 code. At the same time, "Content-Type: plain/text" (without charset) and "Content-Type: plain/text; charset=UTF-8" (in uppercase) were accepted.

      We're running the official Ozone v1.1.0 release.

      These lines shown as below are known behavior:

      • 'text/plain' ==> OK
      • 'text/plain; charset=UTF-8' ==> OK
      • 'text/plain; charset=utf-8' ==> 500
      • 'text/html; charset=UTF-8' ==> OK (same as text/plain case)
      • 'text/html; charset=utf-8' ==> 500 (same as text/plain case)
      • 'text/markdown; charset=UTF-8' ==> OK
      • 'text/markdown; charset=utf-8' ==> OK

      I've attached a code snippet (upload.go) and a log from S3 gateway (s3g.log). This code can run easily with following command lines.

      % go run upload.go --contenttype='text/plain; charset=utf-8'
      text/plain; charset=utf-8
      InternalServerError: Internal Server Error
       status code: 500, request id: , host id: exit status 1
      
      % go run upload.go --contenttype='text/plain; charset=UTF-8'
      text/plain; charset=UTF-8
      &\{http://example.com/<BUCKET>/<KEY> <nil> <nil>}%
      

      HTTP request from upload.go is here:

      PUT /<BUCKET>/<KEY> HTTP/1.1
      Host: localhost:9999
      User-Agent: aws-sdk-go/1.38.46 (go1.16.4; linux; amd64) S3Manager
      Content-Length: 4
      Authorization: AWS4-HMAC-SHA256 Credential=ksugihara@EXAMPLE.COM/20210525/dummy/s3/aws4_request, SignedHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=<HASH>
      Content-Md5: AzvZSxFo1+Tw1kTDyV41vw==
      Content-Type: text/plain; charset=utf-8
      X-Amz-Content-Sha256: <HASH>
      X-Amz-Date: 20210525T080147Z
      Accept-Encoding: gzip
      
      TEST^C
      

      Investigation

      We thought the issue is similar to HDDS-4856 but this issue will happen when empty Content-Type header was given. Our facing problem will appear when Content-Type defined together with the charset directive.

      The stack trace of S3 gateway indicates the request proceeded to remote invocation and failed with authentication error with org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken). There are many parts to preprocess a request before reaching RPC Client, so I suspect some Java servlet filters like ozone.s3.HeaderPreprocessor and hdds.server.http.NoCacheFilter. But as far as I found out, these are not a cause of this problem directly (because these are almost for generating response header). I also suspected some components like OzoneClientProducer, Signature Processor and HeaderPreprocessor. I think this is not direct cause. How do I investigate this problem in more detail?

      Attachments

        1. s3g.log
          53 kB
          Kohei Sugihara
        2. upload.go
          1 kB
          Kohei Sugihara

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            ksugihara Kohei Sugihara

            Dates

              Created:
              Updated:

              Issue deployment