[SPARK-27002] Latest kafka delegation token not always picked up

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 27/Feb/19 10:43
• Updated: 27/Feb/19 18:10
• Resolved: 27/Feb/19 18:07

Description

Lately KAFKA-8004 has been discovered and this triggered me to do further tests with Spark.
Namely I've created long running Kafka to Kafka tests on 4 node cluster with randomly thrown artificial exceptions.

Test scenario:

• 4 node cluster
• Yarn
• Kafka broker version 2.1.0
• security.protocol = SASL_SSL
• sasl.mechanism = SCRAM-SHA-512

Kafka broker settings:

• delegation.token.expiry.time.ms=600000 (10 min)
• delegation.token.max.lifetime.ms=1200000 (20 min)
• delegation.token.expiry.check.interval.ms=300000 (5 min)

After each 7.5 minutes new delegation token obtained from Kafka broker (10 min * 0.75).
But when token expired after 10 minutes (Spark obtains new one and doesn't renew the old), the brokers expiring thread comes after each 5 minutes (invalidates expired tokens) and artificial exception has been thrown inside the Spark application (such case Spark closes connection), then the latest delegation token not always picked up.

Issue Links

• links to: GitHub Pull Request #23906