[SPARK-17924] Consolidate streaming and batch write path

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 13/Oct/16 23:53
• Updated: 30/Jun/17 17:37
• Resolved: 30/Jun/17 17:37

Description

Structured streaming and normal SQL operation currently have two separate write path, leading to a lot of duplicated functions (that look similar) and if branches. The purpose of this ticket is to consolidate the two as much as possible to make the write path more clear.

A side-effect of this change is that streaming will automatically support all the file formats.

Issue Links

• is duplicated by: SPARK-15765 Make continuous Parquet writes consistent with non-continuous Parquet writes
• is related to: SPARK-15472 Add support for writing partitioned `csv`, `json`, `text` formats in Structured Streaming

Sub-Tasks

• Break fileSourceInterfaces.scala into multiple pieces: Resolved
• Remove dead code in WriterContainer: Resolved
• Simplify WriterContainer code: Resolved
• Refactor file name specification for data sources: Resolved
• OutputWriter needs to return the path of the file written: Resolved
• Introduce an internal commit protocol API along with OutputCommitter implementation: Resolved
• Port streaming to use the commit protocol API: Resolved
• Port RDD API to use commit protocol: Resolved
• Support all file formats in structured streaming: Resolved
• Move commit protocol API from sql to core module: Resolved
• Implement commit protocol to support `mapred` package's committer: Resolved
• Move `SparkHadoopWriter` to `internal/io/`: Resolved