Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9688

Improve spark-ec2 script to handle users that are not root

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.4.0, 1.4.1
    • Fix Version/s: None
    • Component/s: EC2
    • Labels:
    • Environment:

      All

    • Flags:
      Important

      Description

      Hi,

      I was trying to use the spark-ec2 script from Spark to create a new Spark cluster wit an user different than root (--user=ec2-user). Unfortunately the part of the script that attempts to copy the templates into the target machines fail because it tries to rsync /etc/* and /root/*

      This is the full traceback

      rsync: recv_generator: mkdir "/root/spark-ec2" failed: Permission denied (13)

          • Skipping any contents from this failed directory ***

      sent 95 bytes received 17 bytes 224.00 bytes/sec
      total size is 1444 speedup is 12.89
      rsync error: some files/attrs were not transferred (see previous errors) (code 2 3) at main.c(1039) [sender=3.0.6]
      Traceback (most recent call last):
      File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1455, in <module>
      main()
      File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1447, in main
      real_main()
      File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1283, in real_main
      setup_cluster(conn, master_nodes, slave_nodes, opts, True)
      File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 785, in setup_cluster
      modules=modules
      File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1049, in deploy_files
      subprocess.check_call(command)
      File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
      raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['rsync', '-rv', '-e', 'ssh -o StrictHos tKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ec2-user/.ssh/sparkclus terkey_us_east.pem', '/tmp/tmpT4Iw54/', u'ec2-user@ec2-52-2-96-193.compute-1.ama zonaws.com:/']' returned non-zero exit status 23

      Is there a workaround for this? I want to improve security of our operations by avoiding user root on the instances.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kuribe Karina Uribe
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 252h
                252h
                Remaining:
                Remaining Estimate - 252h
                252h
                Logged:
                Time Spent - Not Specified
                Not Specified