Bigtop
  1. Bigtop
  2. BIGTOP-846

The helper script introduced in BIGTOP-547 doesn't create /user/$USER directories on HDFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      For BIGTOP-547, I had done my testing where I running the MR job as root.

      However, the helper script creates home directory for hard-coded set of users (root, jenkins, etc.) but not for other users. In particular, it doesn't create a directory for the user who is running the command.

      It should do that too (safely, in case the user is root).

        Issue Links

          Activity

          Hide
          Anatoli Fomenko added a comment -

          Added a new version of the patch that checks if the current user home directory exists, prior to adding (per Mark's comments).

          Show
          Anatoli Fomenko added a comment - Added a new version of the patch that checks if the current user home directory exists, prior to adding (per Mark's comments).
          Hide
          Roman Shaposhnik added a comment -

          A thought (in a form of Bruno pinging me ) has just occurred to me: wouldn't it be better for each component to advertise how exactly it need to initialize the HDFS via a json/xml/txt file dropped into a predetermined location describing exactly what needs to happen. We can them have a very generic script that simply operates on that.

          Show
          Roman Shaposhnik added a comment - A thought (in a form of Bruno pinging me ) has just occurred to me: wouldn't it be better for each component to advertise how exactly it need to initialize the HDFS via a json/xml/txt file dropped into a predetermined location describing exactly what needs to happen. We can them have a very generic script that simply operates on that.
          Hide
          Mark Grover added a comment -

          Anatoli, both the patches appear the same to me. Am I missing something?

          Show
          Mark Grover added a comment - Anatoli, both the patches appear the same to me. Am I missing something?
          Hide
          Mark Grover added a comment -

          Roman, I like that idea. However, I would personally be inclined towards still fixing this issue in this JIRA and working separately on the generic script on a separate JIRA.

          Show
          Mark Grover added a comment - Roman, I like that idea. However, I would personally be inclined towards still fixing this issue in this JIRA and working separately on the generic script on a separate JIRA.
          Hide
          Anatoli Fomenko added a comment -

          Added new patch (thank you Mark)

          Show
          Anatoli Fomenko added a comment - Added new patch (thank you Mark)
          Hide
          Bruno Mahé added a comment -

          For now to fix that jira, can we make the creation of that user optional? Or better, make it passed as a parameter? The latest patch does not seem give such option.
          I don't always want to create a user directory for the user running that script. The user running that script may also not always be the one using Apache Hadoop (for instance when a vm gets created). So being able to pass the user as a parameter would be very nice (ie, creating home dir for various users when creating the vm).

          Also, I am not sure we want to make that directory writable to the group or the world by default.

          Show
          Bruno Mahé added a comment - For now to fix that jira, can we make the creation of that user optional? Or better, make it passed as a parameter? The latest patch does not seem give such option. I don't always want to create a user directory for the user running that script. The user running that script may also not always be the one using Apache Hadoop (for instance when a vm gets created). So being able to pass the user as a parameter would be very nice (ie, creating home dir for various users when creating the vm). Also, I am not sure we want to make that directory writable to the group or the world by default.
          Hide
          Roman Shaposhnik added a comment -

          Bruno, I agree that the default behavior should be like it is today and only if -u uname is specified we should bother with the user creation. I'll update the patch to take care of that (and also the default permissions).

          At the same time, I just don't want us to spend too much time polishing the current script since it is definitely getting replaced in Bigtop 0.7.0 with a more efficient implementation (BIGTOP-952).

          Show
          Roman Shaposhnik added a comment - Bruno, I agree that the default behavior should be like it is today and only if -u uname is specified we should bother with the user creation. I'll update the patch to take care of that (and also the default permissions). At the same time, I just don't want us to spend too much time polishing the current script since it is definitely getting replaced in Bigtop 0.7.0 with a more efficient implementation ( BIGTOP-952 ).
          Hide
          Mark Grover added a comment -

          +1 (non-committer) to latest patch

          Show
          Mark Grover added a comment - +1 (non-committer) to latest patch
          Hide
          Bruno Mahé added a comment -

          Thanks!
          I was only asking for pushing the user into a parameter and that's exactly what the latest patch provides.

          Patch looks good.
          +1

          Show
          Bruno Mahé added a comment - Thanks! I was only asking for pushing the user into a parameter and that's exactly what the latest patch provides. Patch looks good. +1

            People

            • Assignee:
              Anatoli Fomenko
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development