[SPARK-26121] [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Anastasios Zouzias
• Reporter: Anastasios Zouzias
• Shepherd: Cody Koeninger
• Created: 19/Nov/18 20:20
• Updated: 26/Nov/18 17:15
• Resolved: 26/Nov/18 17:15

Description

I run in the following situation with Spark Structure Streaming (SS) using Kafka.
 
In a project that I work on, there is already a secured Kafka setup where ops can issue an SSL certificate per "group.id", which should be predefined (or its prefix to be predefined).
 
On the other hand, Spark SS fixes the group.id to 
 
val uniqueGroupId = s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
 
see, i.e.,
 
https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124
 
https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
 
I guess Spark developers had a good reason to fix it, but is it possible to make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
 
The rational is that spark users are not forced to use the same certificate on group-ids of the form (spark-kafka-source-*).
 
DoD:

• Allow spark SS users to define the group.id prefix as input parameter.

Issue Links

• links to: [Github] Pull Request #23103 (zouzias)