[SPARK-9711] Unable to run spark after restarting cluster with spark-ec2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.4.1
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Guangyang Li
• Created: 06/Aug/15 22:26
• Updated: 09/Feb/16 09:15
• Resolved: 09/Feb/16 09:15

Description

With Spark 1.4.1 and YARN client mode, my application works at the first time the cluster is built. While if I stop and start the cluster with using spark-ec2, the same command fails. At the end of the spark logs, it's shown that it just keeps trying to connect to master node repeatedly:

INFO Client: Retrying connect to server: ec2-54-174-232-129.compute-1.amazonaws.com/172.31.36.29:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

I restarted YARN and dfs manually after restarting the cluster, however, I was unable to restart Tachyon and it fails when running ./bin/tachyon runTests, which might be the possible reason.