[SPARK-20703] Add an operator for writing data out

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: Reynold Xin
• Created: 11/May/17 01:28
• Updated: 17/Aug/17 15:58
• Resolved: 06/Jul/17 07:47

Description

We should add an operator for writing data out. Right now in the explain plan / UI there is no way to tell whether a query is writing data out, and also there is no way to associate metrics with data writes. It'd be tremendously valuable to do this for adding metrics and for visibility.

Issue Links

• is related to: SPARK-21762 FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible
• links to: [Github] Pull Request #17998 (viirya)
• links to: [Github] Pull Request #18159 (viirya)
• links to: [Github] Pull Request #18558 (viirya)