[SPARK-41551] Improve/complete PathOutputCommitProtocol support for dynamic partitioning

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 16/Dec/22 17:45
• Updated: 28/Feb/23 16:16

Description

Followup to SPARK-40034 as

• that is incomplete as it doesn't record the partitions
• as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow renames are ok, both s3a committers are actually OK to use.

It's only that newTaskTempFileAbsPath operation which is unsupported in s3a committers; the post-job dir rename is O(data) but file by file rename is correct for a non-atomic job commit.

1. Cut PathOutputCommitProtocol.newTaskTempFile; to update super partitionPaths (needs a setter). The superclass can't just say if (committer instance of PathOutputCommitter as spark-core needs to compile with older hadoop versions)
2. downgrade failure in setup to log (info?)
3. retain failure in the newTaskTempFileAbsPath call.

Testing: yes

Issue Links

• requires: SPARK-40034 PathOutputCommitters to work with dynamic partition overwrite
• links to: [Github] Pull Request #39185 (steveloughran)
• links to: [Github] Pull Request #40221 (steveloughran)
• links to: [Github] Pull Request #40221 (steveloughran)