[KUDU-2539] Supporting Spark Streaming DataFrame in KuduContext

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0
• Component/s: spark
• Code Review: https://gerrit.cloudera.org/#/c/11199/
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 13/Aug/18 10:27
• Updated: 26/Sep/18 15:47
• Resolved: 26/Sep/18 15:47

Description

Currently KuduContext does not support Spark Streaming DataFrame. The problem comes from a foreachPartition call which in case of spark streaming is an unsupported operation, like foreach: 

unsupported operations in streaming

I have created a small example app with a custom Kudu sink which can be used for testing:

kudu custom sink and example app

The patch fixing this issue is also available for kudu-spark, so soon a gerrit review can be expected with the solution.