[SPARK-23539] Add support for Kafka headers in Structured Streaming

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Dongjin Lee
• Reporter: Tathagata Das
• Created: 28/Feb/18 21:35
• Updated: 27/Oct/20 11:32
• Resolved: 13/Sep/19 17:32

Description

Kafka headers were added in 0.11. We should expose them through our kafka data source in both batch and streaming queries. 

This is currently blocked on version of Kafka in Spark from 0.10.1 to 1.0+ SPARK-18057

Issue Links

• contains: SPARK-28446 Document Kafka Headers support
• is blocked by: SPARK-18057 Update structured streaming kafka from 0.10.0.1 to 2.0.0
• links to: [Github] Pull Request #22282 (dongjinleekr)
• links to: GitHub Pull Request #22282
• links to: GitHub Pull Request #25792