[SPARK-11831] AkkaRpcEnvSuite is prone to port-contention-related flakiness

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: Tests
• Labels: flaky-test
• Target Version/s: 1.5.3, 1.6.0
• Assignee: Shixiong Zhu
• Reporter: Josh Rosen
• Created: 18/Nov/15 22:55
• Updated: 20/Nov/15 00:52
• Resolved: 19/Nov/15 22:50

Description

The AkkaRpcEnvSuite tests appear to be prone to port-contention-related flakiness in Jenkins:

Error Message

Failed to bind to: localhost/127.0.0.1:12362: Service 'test' failed after 16 retries!
Stacktrace

      java.net.BindException: Failed to bind to: localhost/127.0.0.1:12362: Service 'test' failed after 16 retries!
      at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
      at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
      at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
      at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
      at scala.util.Try$.apply(Try.scala:161)
      at scala.util.Success.map(Try.scala:206)
      at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
      at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
      at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
      at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
      at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
      at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
      at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
      at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
      at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Test/job/Spark-Master-Maven-pre-YARN/4819/HADOOP_VERSION=1.2.1,label=spark-test/testReport/junit/org.apache.spark.rpc.akka/AkkaRpcEnvSuite/uriOf__ssl/

We should probably refactor these tests to not depend on a fixed port.

Issue Links

• links to: [Github] Pull Request #9841 (zsxwing)
• links to: [Github] Pull Request #9850 (zsxwing)