[SPARK-30442] Write mode ignored when using CodecStreams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.4
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Jesse Collins
• Created: 06/Jan/20 20:29
• Updated: 27/Feb/20 22:49
• Resolved: 23/Jan/20 02:29

Description

Overwrite is hardcoded to false in the codec stream. This can cause issues, particularly with aws tools, that make it impossible to retry.

Ideally, this should be read from the write mode set for the DataWriter that is writing through this codec class.

https://github.com/apache/spark/blame/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala#L81