[SPARK-40034] PathOutputCommitters to work with dynamic partition overwrite

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core, SQL
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Aug/22 14:24
• Updated: 01/Mar/23 09:55
• Resolved: 09/Sep/22 18:29

Description

sibling of MAPREDUCE-7403: allow PathOutputCommitter implementation to declare that they support the semantics required by spark dynamic partitioning:

• rename to work as expected
• working dir to be on same fs as final dir

They will do this through implementing StreamCapabilities and adding a new probe, "mapreduce.job.committer.dynamic.partitioning" ; the spark side changes are to

• postpone rejection of dynamic partition overwrite until the output committer is created
• allow it if the committer implements StreamCapabilities and returns true for {{hasCapability("mapreduce.job.committer.dynamic.partitioning")))

this isn't going to be supported by the s3a committers, they don't meet the requirements. The manifest committer of MAPREDUCE-7341 running against abfs and gcs does work.

Issue Links

• is required by: SPARK-41551 Improve/complete PathOutputCommitProtocol support for dynamic partitioning
• relates to: MAPREDUCE-7403 Support spark dynamic partitioning in the Manifest Committer
• relates to: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• links to: [Github] Pull Request #37468 (steveloughran)