[HADOOP-17112] whitespace not allowed in paths when saving files to s3a via committer

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.1, 3.4.0, 3.2.3
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.1
• Assignee: Krzysztof Adamski
• Reporter: Krzysztof Adamski
• Created: 03/Jul/20 14:08
• Updated: 28/Feb/22 01:43
• Resolved: 25/Apr/21 18:00

Description

When saving results through spark dataframe on latest 3.0.1-snapshot compiled against hadoop-3.2 with the following specs
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
--conf spark.hadoop.fs.s3a.committer.name=partitioned
--conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
we are unable to save the file with whitespace character in the path. It works fine without.

I was looking into the recent commits with regards to qualifying the path, but couldn't find anything obvious. Is this a known bug?

When saving results through spark dataframe on latest 3.0.1-snapshot compiled against hadoop-3.2 with the following specs
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
--conf spark.hadoop.fs.s3a.committer.name=partitioned
--conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
we are unable to save the file with whitespace character in the path. It works fine without.

I was looking into the recent commits with regards to qualifying the path, but couldn't find anything obvious. Is this a known bug?

Issue Links

• links to: GitHub Pull Request #2953