[SPARK-29054] Invalidate Kafka consumer when new delegation token available

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 11/Sep/19 12:32
• Updated: 03/Oct/19 16:35
• Resolved: 03/Oct/19 16:35

Description

Kafka consumers are cached. If delegation token is used and the token is expired, then exception is thrown. Such case new consumer is created in a Task retry with the latest delegation token. This can be enhanced by detecting the existence of a new delegation token.

Issue Links

• links to: GitHub Pull Request #25760