[SPARK-24396] Add Structured Streaming ForeachWriter for python

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Target Version/s: 2.4.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 25/May/18 23:57
• Updated: 13/Jul/18 05:26
• Resolved: 15/Jun/18 19:57

Description

Users should be able to write ForeachWriter code in python, that is, they should be able to use the partitionid and the version/batchId/epochId to conditionally process rows.

Issue Links

• is blocked by: SPARK-24397 Add TaskContext.getLocalProperties in Python
• links to: [Github] Pull Request #21477 (tdas)