[ARROW-15306] [C++] S3FileSystem Should set the content-type header to application/octet-stream if not specified

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30796
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 11/Jan/22 20:28
• Updated: 11/Jan/23 11:36
• Resolved: 12/Jan/22 10:37

Description

By default the S3FileSystem leaves the header alone, which is technically correct as I don't think a content-type should be specified if one isn't known.

However, the aws-s3-sdk appears to set the content-type to application/xml whenever it is not specified: https://github.com/aws/aws-sdk-cpp/blob/5378016f845fe85e334ffc30319614e7d4dad41f/aws-cpp-sdk-s3/include/aws/s3/S3Request.h#L41

We could potentially file an issue with the S3 SDK but I'm not sure if that would make any progress (and S3 itself may require a content-type always be present for some reason).

Since there is no way to avoid specifying a content-type then we should default to application/octet-stream which is a more accurate "I don't know what this file is" than "application/xml".

The content-type can confuse libraries that try and automatically act on the file based on the content-type. See https://github.com/apache/arrow/issues/11934

Issue Links

• links to: GitHub Pull Request #12129