[SPARK-20597] KafkaSourceProvider falls back on path as synonym for topic

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Structured Streaming
• Labels: bulk-closed starter
• Assignee: Unassigned
• Reporter: Jacek Laskowski
• Shepherd: Michael Armbrust
• Created: 04/May/17 12:11
• Updated: 25/May/21 01:52
• Resolved: 25/May/21 01:40

Description

1. KafkaSourceProvider supports topic option that sets the Kafka topic to save a DataFrame's rows to
2. KafkaSourceProvider can use topic column to assign rows to Kafka topics for writing

What seems a quite interesting option is to support start(path: String) as the least precedence option in which path would designate the default topic when no other options are used.

df.writeStream.format("kafka").start("topic")

See http://apache-spark-developers-list.1001551.n3.nabble.com/KafkaSourceProvider-Why-topic-option-and-column-without-reverting-to-path-as-the-least-priority-td21458.html for discussion

Issue Links

• links to: GitHub Pull Request #23791