[SPARK-35383] Improve s3a magic committer support by inferring missing configs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 12/May/21 05:37
• Updated: 12/May/21 18:53
• Resolved: 12/May/21 18:53

Description

Exception in thread "main" org.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://my-spark-bucket`: Filesystem does not have support for 'magic' committer enabled in configuration option fs.s3a.committer.magic.enabled
	at org.apache.hadoop.fs.s3a.commit.CommitUtils.verifyIsMagicCommitFS(CommitUtils.java:74)
	at org.apache.hadoop.fs.s3a.commit.CommitUtils.getS3AFileSystem(CommitUtils.java:109)

Issue Links

• links to: [Github] Pull Request #32518 (dongjoon-hyun)