[SPARK-30541] Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL, Structured Streaming
• Target Version/s: 3.0.0
• Assignee: Gabor Somogyi
• Reporter: Jungtaek Lim
• Created: 17/Jan/20 07:42
• Updated: 24/Mar/20 08:36
• Resolved: 22/Mar/20 02:00

Description

The test suite has been failing intermittently as of now:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116862/testReport/

 

org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite.(It is not a test it is a sbt.testing.SuiteSelector)
 

Error Details
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 3939 times over 1.0001223535333332 minutes. Last failure message: KeeperErrorCode = AuthFailed for /brokers/ids.

Stack Trace
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 3939 times over 1.0001223535333332 minutes. Last failure message: KeeperErrorCode = AuthFailed for /brokers/ids.
	at org.scalatest.concurrent.Eventually.tryTryAgain$1(Eventually.scala:432)
	at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:439)
	at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:391)
	at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:479)
	at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:337)
	at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:336)
	at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:479)
	at org.apache.spark.sql.kafka010.KafkaTestUtils.setup(KafkaTestUtils.scala:292)
	at org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite.beforeAll(KafkaDelegationTokenSuite.scala:49)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:58)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:317)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:510)
	at sbt.ForkMain$Run$2.call(ForkMain.java:296)
	at sbt.ForkMain$Run$2.call(ForkMain.java:286)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: sbt.ForkMain$ForkError: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /brokers/ids
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at kafka.zookeeper.AsyncResponse.resultException(ZooKeeperClient.scala:554)
	at kafka.zk.KafkaZkClient.getChildren(KafkaZkClient.scala:719)
	at kafka.zk.KafkaZkClient.getSortedBrokerList(KafkaZkClient.scala:455)
	at kafka.zk.KafkaZkClient.getAllBrokersInCluster(KafkaZkClient.scala:404)
	at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setup$3(KafkaTestUtils.scala:293)
	at org.scalatest.concurrent.Eventually.makeAValiantAttempt$1(Eventually.scala:395)
	at org.scalatest.concurrent.Eventually.tryTryAgain$1(Eventually.scala:409)
	... 20 more

Issue Links

• causes: SPARK-31050 Disable flaky KafkaDelegationTokenSuite
• links to: GitHub Pull Request #27259
• links to: GitHub Pull Request #27789
• links to: GitHub Pull Request #27877