Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-563 Security features for Map/Reduce
  3. MAPREDUCE-1457

For secure job execution, couple of more UserGroupInformation.doAs needs to be added

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During our testing in a kerberos environment, we had to add UserGroupInformation.doAs blocks in certain places.

      1. MAPREDUCE-1457-BPY20.patch
        4 kB
        Jakob Homan
      2. MAPREDUCE-1457.patch
        7 kB
        Jakob Homan
      3. MAPREDUCE-1457-BPY20.patch
        11 kB
        Jakob Homan
      4. MAPREDUCE-1457.patch
        8 kB
        Jakob Homan
      5. MAPREDUCE-1457.patch
        8 kB
        Jakob Homan
      6. MAPREDUCE-1457-BPY20.patch.1
        10 kB
        Devaraj Das

        Activity

        Hide
        Jakob Homan added a comment -

        Attaching patch for Y!'s 20 distribution.

        Show
        Jakob Homan added a comment - Attaching patch for Y!'s 20 distribution.
        Hide
        Jakob Homan added a comment -

        Attaching patch for Apache trunk.

        Show
        Jakob Homan added a comment - Attaching patch for Apache trunk.
        Hide
        Jakob Homan added a comment -

        Submitting patch.

        Show
        Jakob Homan added a comment - Submitting patch.
        Hide
        Devaraj Das added a comment -

        Some comments:
        In the trunk patch and in the Y20 patch, the call to task.taskCleanup in Child.java should be within a privileged block.
        In the Y20 patch, the calls to JobHistory.initDone() and JobHistory.init() should be within privileged blocks, and use the JobTracker's ugi. Within JobHistory.logSubmitted, there is a call to set up the writer for userLogFile. That should be within a privileged block and use the user's ugi.

        Show
        Devaraj Das added a comment - Some comments: In the trunk patch and in the Y20 patch, the call to task.taskCleanup in Child.java should be within a privileged block. In the Y20 patch, the calls to JobHistory.initDone() and JobHistory.init() should be within privileged blocks, and use the JobTracker's ugi. Within JobHistory.logSubmitted, there is a call to set up the writer for userLogFile. That should be within a privileged block and use the user's ugi.
        Hide
        Jakob Homan added a comment -

        Updated Y! 20 patch, per Devaraj's comments.

        Show
        Jakob Homan added a comment - Updated Y! 20 patch, per Devaraj's comments.
        Hide
        Jakob Homan added a comment -

        Updated trunk patch. Removes a couple unneeded variables.

        Show
        Jakob Homan added a comment - Updated trunk patch. Removes a couple unneeded variables.
        Hide
        Jakob Homan added a comment -

        Submitting patch.

        Show
        Jakob Homan added a comment - Submitting patch.
        Hide
        Devaraj Das added a comment -

        +1

        Show
        Devaraj Das added a comment - +1
        Hide
        Jakob Homan added a comment -

        String for user should be out of doAs block...

        Show
        Jakob Homan added a comment - String for user should be out of doAs block...
        Hide
        Devaraj Das added a comment -

        +1 (also ran tests/test-patch manually). I will commit this.

        Show
        Devaraj Das added a comment - +1 (also ran tests/test-patch manually). I will commit this.
        Hide
        Devaraj Das added a comment -

        Regarding addition of new tests, the bugs fixed by the patch shows up in only the secure environment (and hence difficult to write testcases). The patch was manually tested.

        Show
        Devaraj Das added a comment - Regarding addition of new tests, the bugs fixed by the patch shows up in only the secure environment (and hence difficult to write testcases). The patch was manually tested.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Can you please explain the individual changes for the sake for records? Couldn't see what this issue is addressing without looking at the patch itself. Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - Can you please explain the individual changes for the sake for records? Couldn't see what this issue is addressing without looking at the patch itself. Thanks!
        Hide
        Devaraj Das added a comment -

        Ok the individual changes:
        1) In the JobTracker, the getStagingArea RPC needs to construct a path for the user to write job files to. The getStagingArea does a getFileSystem and internally the getFileSystem sets up a connection to the namenode. For this connection, the JobTracker's credential should be used. That's why the mrOwner.doAs in that method is required.
        2) In Child.java, the task authenticates to the TaskTracker using the jobtoken. The username in the jobtoken is jobId. The doAs block done using taskOwner is required so that the username mentioned in the token and the one doing the operation matches.
        3) In Child.java, the task execution and the task cleanup are within doAs blocks and those doAs blocks are run as the user submitting the job. In the former part, the task communicates with the namenode, and in the latter, it could potentially communicate with the namenode (abortTask creates a connection to the namenode, etc). These are within doAs blocks so that the username mentioned in the delegation token (the job submitting user) matches with the user performing the operation.

        Show
        Devaraj Das added a comment - Ok the individual changes: 1) In the JobTracker, the getStagingArea RPC needs to construct a path for the user to write job files to. The getStagingArea does a getFileSystem and internally the getFileSystem sets up a connection to the namenode. For this connection, the JobTracker's credential should be used. That's why the mrOwner.doAs in that method is required. 2) In Child.java, the task authenticates to the TaskTracker using the jobtoken. The username in the jobtoken is jobId. The doAs block done using taskOwner is required so that the username mentioned in the token and the one doing the operation matches. 3) In Child.java, the task execution and the task cleanup are within doAs blocks and those doAs blocks are run as the user submitting the job. In the former part, the task communicates with the namenode, and in the latter, it could potentially communicate with the namenode (abortTask creates a connection to the namenode, etc). These are within doAs blocks so that the username mentioned in the delegation token (the job submitting user) matches with the user performing the operation.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Jakob!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Jakob!
        Hide
        Devaraj Das added a comment -

        Patch for Y20. Not to be committed here.

        Show
        Devaraj Das added a comment - Patch for Y20. Not to be committed here.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #228 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/228/)
        . Fixes JobTracker to get the FileSystem object within getStagingAreaDir within a privileged block. Fixes Child.java to use the appropriate UGIs while getting the TaskUmbilicalProtocol proxy and while executing the task. Contributed by Jakob Homan.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #228 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/228/ ) . Fixes JobTracker to get the FileSystem object within getStagingAreaDir within a privileged block. Fixes Child.java to use the appropriate UGIs while getting the TaskUmbilicalProtocol proxy and while executing the task. Contributed by Jakob Homan.

          People

          • Assignee:
            Jakob Homan
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development