[SPARK-32566] kafka consumer cache capacity is unclear

• Type: Documentation
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: srpn
• Created: 07/Aug/20 11:04
• Updated: 10/Aug/20 03:10
• Resolved: 10/Aug/20 03:09

Description

The docs mention

The cache for consumers has a default maximum size of 64.  If you expect
 to be handling more than (64 * number of executors) Kafka partitions, 
you can change this setting via spark.streaming.kafka.consumer.cache.maxCapacity

However, for structured streaming, the code seems to expect

spark.kafka.consumer.cache.capacity/spark.sql.kafkaConsumerCache.capacity

Would be nice to clear the ambiguity in the documentation or even merge these configurations in the code