[SPARK-926] spark_ec2 script when ssh/scp-ing should pipe UserknowHostFile to /dev/null

• Type: New Feature
• Status: Resolved
• Priority: Trivial
• Resolution: Duplicate
• Affects Version/s: 0.8.0
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Shay Seng
• Created: 08/Oct/13 11:03
• Updated: 06/Feb/15 07:34
• Resolved: 06/Feb/15 07:34

Description

The know host file in the local machine gets all kinds of crap after a few cluster launches. When SSHing, or SCPing, please add "-o UserKnowHostFile=/dev/null"

Also remove the -t option from SSH, and only add in when necessary - to reduce chatter on console.
e.g.

1. Copy a file to a given host through scp, throwing an exception if scp fails
   def scp(host, opts, local_file, dest_file):
   subprocess.check_call(
   "scp -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i %s '%s' '%s@%s:%s'" %
   (opts.identity_file, local_file, opts.user, host, dest_file), shell=True)

1. Run a command on a host through ssh, retrying up to two times
2. and then throwing an exception if ssh continues to fail.
   def ssh(host, opts, command, sshopts=""):
   tries = 0
   while True:
   try:
3. removed -t option from ssh command, not sure why it is required all the time.
   return subprocess.check_call(
   "ssh %s -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i %s %s@%s '%s'" %
   (sshopts, opts.identity_file, opts.user, host, command), shell=True)
   except subprocess.CalledProcessError as e:
   if (tries > 2):
   raise e
   print "Couldn't connect to host {0}

   , waiting 30 seconds".format(e)
   time.sleep(30)
   tries = tries + 1

Issue Links

• duplicates: SPARK-5403 Ignore UserKnownHostsFile in SSH calls
• is duplicated by: SPARK-5403 Ignore UserKnownHostsFile in SSH calls