When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size to 5 Mb, storing data in AWS doesn't work anymore. For example, running the following code:
shows the following exception:
After some investigation, I discovered that hadoop-aws doesn't send SSE-C headers in Put Object Part as stated in AWS specification: https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
You can find a patch attached to this issue for a better clarification of the problem.