Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1385

Make changes to MapReduce for the new UserGroupInformation APIs (HADOOP-6299)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      This is about moving the MapReduce code to use the new UserGroupInformation API as described in HADOOP-6299.

      1. mr-6299.8.patch
        160 kB
        Devaraj Das
      2. mr-6299.7.patch
        159 kB
        Devaraj Das
      3. mr-6299.3.patch
        155 kB
        Devaraj Das
      4. mr-6299.patch
        143 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          The patch for trunk

          Show
          Devaraj Das added a comment - The patch for trunk
          Hide
          Devaraj Das added a comment -

          The earlier patch went out of sync. Updated patch.

          Show
          Devaraj Das added a comment - The earlier patch went out of sync. Updated patch.
          Hide
          Owen O'Malley added a comment -

          Please add a call to UserGroupInformation.setConfiguration at the start of the JobTracker, and TaskTracker so that UGI uses the same configuration object. The Task should also use setConfiguration with the job's configuration.

          Show
          Owen O'Malley added a comment - Please add a call to UserGroupInformation.setConfiguration at the start of the JobTracker, and TaskTracker so that UGI uses the same configuration object. The Task should also use setConfiguration with the job's configuration.
          Hide
          Owen O'Malley added a comment -

          Please use the long form of the user name rather than the short form in spots like:

          -    JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getUserName()), 
          +    JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getShortUserName()), 
          
          Show
          Owen O'Malley added a comment - Please use the long form of the user name rather than the short form in spots like: - JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getUserName()), + JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getShortUserName()),
          Hide
          Owen O'Malley added a comment -

          The following code from Cluster.java and ClusterWithLinuxTaskController is a no-op:

          UserGroupInformation.getCurrentUser().doAs(...)
          
          Show
          Owen O'Malley added a comment - The following code from Cluster.java and ClusterWithLinuxTaskController is a no-op: UserGroupInformation.getCurrentUser().doAs(...)
          Hide
          Devaraj Das added a comment -

          Patch that addresses Owen's concerns. Running tests now. Running test-patch proved to be a real pain. Still trying to run it..

          Show
          Devaraj Das added a comment - Patch that addresses Owen's concerns. Running tests now. Running test-patch proved to be a real pain. Still trying to run it..
          Hide
          Devaraj Das added a comment -

          Tests pass with this patch.

          Show
          Devaraj Das added a comment - Tests pass with this patch.
          Hide
          Devaraj Das added a comment -

          Ok i could run findbugs on this patch and confirmed that it doesn't add any new warnings. I also ran javadoc and javac-warnings ones. No new warnings there too. I am sure I didn't introduce any new release audit warnings since there are no new files in the patch. Same for @author tags. It is almost impossible to run the stand-alone test-patch command, as it exists today, on this patch.

          Show
          Devaraj Das added a comment - Ok i could run findbugs on this patch and confirmed that it doesn't add any new warnings. I also ran javadoc and javac-warnings ones. No new warnings there too. I am sure I didn't introduce any new release audit warnings since there are no new files in the patch. Same for @author tags. It is almost impossible to run the stand-alone test-patch command, as it exists today, on this patch.
          Hide
          Owen O'Malley added a comment -

          +1

          Show
          Owen O'Malley added a comment - +1
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Devaraj!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Devaraj!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/255/)
          MAPREDUCE-1421. LinuxTaskController tests failing on trunk after the commit of . Contributed by Amareshwari Sriramadasu.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/255/ ) MAPREDUCE-1421 . LinuxTaskController tests failing on trunk after the commit of . Contributed by Amareshwari Sriramadasu.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #248 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/248/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #248 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/248/ )

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development