Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1167

got "Permission denied" when creating vagrant home folder in provision.sh

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: deployment, vm
    • Labels:
      None

      Description

      By default, vagrant shell provisioner run shell scripts with superuser privileges. Since provision.sh do hdfs write operations by root, it get permission denied.
      There're two ways we can fix it.
      1). Create vagrant's home folder by hdfs.
      2). Let vagrant shell provisioner run the shell script without superuser privileges. (set privileged to false in Vagrantfile)

      It seems solution 2 is much more elegant.
      Unfortunately, the privileged setting is only available with vagrant 1.3.0+, and there's another issue coming up with 1.3.5+ which failed the network setting in fedora. So, I might suggest to take solution 1 first.

      1. BIGTOP-1167.2.patch
        2 kB
        Evans Ye
      2. BIGTOP-1167.1.patch
        2 kB
        Evans Ye

        Activity

        Hide
        evans_ye Evans Ye added a comment - - edited

        Submit patches for solution 1&2. Still open for discussing.
        Patch file 1167.1 is for solution 1 and 1167.2 is for solution 2.
        For solution 2 it might only work with vagrant 1.3.0-1.3.4 only.

        Show
        evans_ye Evans Ye added a comment - - edited Submit patches for solution 1&2. Still open for discussing. Patch file 1167.1 is for solution 1 and 1167.2 is for solution 2. For solution 2 it might only work with vagrant 1.3.0-1.3.4 only.
        Hide
        jayunit100 jay vyas added a comment -

        Hi evans. Thanks for flagging this And thanks for the patch also ! will test it asap and let you know if it works for me.

        • vagrant 1.27 : bug
        • vagrant 1.3.3 : no bug
        • vagrant 1.4 : bug

        Any thought on why my vagrant instance, 1.3.3 is immune to this problem (FYI im using VBox 4.2.18) ?

        Show
        jayunit100 jay vyas added a comment - Hi evans. Thanks for flagging this And thanks for the patch also ! will test it asap and let you know if it works for me. vagrant 1.27 : bug vagrant 1.3.3 : no bug vagrant 1.4 : bug Any thought on why my vagrant instance, 1.3.3 is immune to this problem (FYI im using VBox 4.2.18) ?
        Hide
        evans_ye Evans Ye added a comment -

        Ok, Jay, I am sorry that I did not make myself clear. The bug itself has no connection with the version of vagrant.
        Let me explain this more clearly. The problem here is that we'll get following error message when provisioning with provision.sh:

        ...
        mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
        ...
        Job Finished in 23.258 seconds
        Estimated value of Pi is 4.00000000000000000000
        

        Which is caused by executing hdfs write operations by root.
        So what I'm trying to do in those patches is to clarify the shell executor and makes user specific commands being executed successfully.
        The vagrant version is only related to the solutions because we can run the shell provisioner either by root or by vagrant.

        I've tested vagrant 1.3.3, and I still got "Permission denied" when creating the folder. So I think the question you're asking is because of the unclear description on the bug.
        Please let me know if there is something still not explained well.
        Thanks for your rapid response

        Show
        evans_ye Evans Ye added a comment - Ok, Jay, I am sorry that I did not make myself clear. The bug itself has no connection with the version of vagrant. Let me explain this more clearly. The problem here is that we'll get following error message when provisioning with provision.sh: ... mkdir: Permission denied: user=root, access=WRITE, inode= "/user" :hdfs:supergroup:drwxr-xr-x ... Job Finished in 23.258 seconds Estimated value of Pi is 4.00000000000000000000 Which is caused by executing hdfs write operations by root. So what I'm trying to do in those patches is to clarify the shell executor and makes user specific commands being executed successfully. The vagrant version is only related to the solutions because we can run the shell provisioner either by root or by vagrant. I've tested vagrant 1.3.3, and I still got "Permission denied" when creating the folder. So I think the question you're asking is because of the unclear description on the bug. Please let me know if there is something still not explained well. Thanks for your rapid response
        Hide
        jayunit100 jay vyas added a comment -

        To be clear , the bug you are getting is here:

               mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
        
        

        1) Ahh okay, so i see : ROOT cannot write to directory /user . I wonder why? Is that because by default only HDFS can make new directories in dir "user" ?

        2) I tested Patch one and IT WORKS Thanks Can a commiter weigh in and push this through I've tested it and it definetly fixes the error.. maybe Peter Linnell ?

        Show
        jayunit100 jay vyas added a comment - To be clear , the bug you are getting is here: mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x 1) Ahh okay, so i see : ROOT cannot write to directory /user . I wonder why? Is that because by default only HDFS can make new directories in dir "user" ? 2) I tested Patch one and IT WORKS Thanks Can a commiter weigh in and push this through I've tested it and it definetly fixes the error.. maybe Peter Linnell ?
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 on patch #1 and committed. Thanks for investigating!

        Show
        rvs Roman Shaposhnik added a comment - +1 on patch #1 and committed. Thanks for investigating!
        Hide
        evans_ye Evans Ye added a comment -

        hi Jay
        Since the namenode was started by hdfs user, the superuser privilige went to hdfs. permissions guide
        You can see that the SVC_USER has been set to "hdfs" in the namenode daemon script(/etc/init.d/hadoop-hdfs-namenode) which is used to determine who is going to run the namenode.

        SVC_USER="hdfs"
        ...
        su -s /bin/bash $SVC_USER -c "cd $WORKING_DIR && $EXEC_PATH --config '$CONF_DIR' start $DAEMON_FLAGS"
        
        Show
        evans_ye Evans Ye added a comment - hi Jay Since the namenode was started by hdfs user, the superuser privilige went to hdfs. permissions guide You can see that the SVC_USER has been set to "hdfs" in the namenode daemon script(/etc/init.d/hadoop-hdfs-namenode) which is used to determine who is going to run the namenode. SVC_USER= "hdfs" ... su -s /bin/bash $SVC_USER -c "cd $WORKING_DIR && $EXEC_PATH --config '$CONF_DIR' start $DAEMON_FLAGS"
        Hide
        jayunit100 jay vyas added a comment -

        got it . thanks.

        +1 for this patch ! lets get it in soon - its tested by 2 people and essential for proper provisioning .

        Show
        jayunit100 jay vyas added a comment - got it . thanks. +1 for this patch ! lets get it in soon - its tested by 2 people and essential for proper provisioning .

          People

          • Assignee:
            rvs Roman Shaposhnik
            Reporter:
            evans_ye Evans Ye
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development