Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1385

Make changes to MapReduce for the new UserGroupInformation APIs (HADOOP-6299)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      This is about moving the MapReduce code to use the new UserGroupInformation API as described in HADOOP-6299.

      1. mr-6299.patch
        143 kB
        Devaraj Das
      2. mr-6299.8.patch
        160 kB
        Devaraj Das
      3. mr-6299.7.patch
        159 kB
        Devaraj Das
      4. mr-6299.3.patch
        155 kB
        Devaraj Das

        Issue Links

          Activity

          Devaraj Das created issue -
          Hide
          Devaraj Das added a comment -

          The patch for trunk

          Show
          Devaraj Das added a comment - The patch for trunk
          Devaraj Das made changes -
          Field Original Value New Value
          Attachment mr-6299.patch [ 12430663 ]
          Devaraj Das made changes -
          Link This issue is blocked by HADOOP-6299 [ HADOOP-6299 ]
          Devaraj Das made changes -
          Link This issue is blocked by HDFS-905 [ HDFS-905 ]
          Hide
          Devaraj Das added a comment -

          The earlier patch went out of sync. Updated patch.

          Show
          Devaraj Das added a comment - The earlier patch went out of sync. Updated patch.
          Devaraj Das made changes -
          Attachment mr-6299.3.patch [ 12430816 ]
          Hide
          Owen O'Malley added a comment -

          Please add a call to UserGroupInformation.setConfiguration at the start of the JobTracker, and TaskTracker so that UGI uses the same configuration object. The Task should also use setConfiguration with the job's configuration.

          Show
          Owen O'Malley added a comment - Please add a call to UserGroupInformation.setConfiguration at the start of the JobTracker, and TaskTracker so that UGI uses the same configuration object. The Task should also use setConfiguration with the job's configuration.
          Hide
          Owen O'Malley added a comment -

          Please use the long form of the user name rather than the short form in spots like:

          -    JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getUserName()), 
          +    JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getShortUserName()), 
          
          Show
          Owen O'Malley added a comment - Please use the long form of the user name rather than the short form in spots like: - JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getUserName()), + JobInfo jobInfo = new JobInfo(jobId, new Text(ugi.getShortUserName()),
          Hide
          Owen O'Malley added a comment -

          The following code from Cluster.java and ClusterWithLinuxTaskController is a no-op:

          UserGroupInformation.getCurrentUser().doAs(...)
          
          Show
          Owen O'Malley added a comment - The following code from Cluster.java and ClusterWithLinuxTaskController is a no-op: UserGroupInformation.getCurrentUser().doAs(...)
          Hide
          Devaraj Das added a comment -

          Patch that addresses Owen's concerns. Running tests now. Running test-patch proved to be a real pain. Still trying to run it..

          Show
          Devaraj Das added a comment - Patch that addresses Owen's concerns. Running tests now. Running test-patch proved to be a real pain. Still trying to run it..
          Devaraj Das made changes -
          Attachment mr-6299.7.patch [ 12431449 ]
          Hide
          Devaraj Das added a comment -

          Tests pass with this patch.

          Show
          Devaraj Das added a comment - Tests pass with this patch.
          Devaraj Das made changes -
          Attachment mr-6299.8.patch [ 12431472 ]
          Hide
          Devaraj Das added a comment -

          Ok i could run findbugs on this patch and confirmed that it doesn't add any new warnings. I also ran javadoc and javac-warnings ones. No new warnings there too. I am sure I didn't introduce any new release audit warnings since there are no new files in the patch. Same for @author tags. It is almost impossible to run the stand-alone test-patch command, as it exists today, on this patch.

          Show
          Devaraj Das added a comment - Ok i could run findbugs on this patch and confirmed that it doesn't add any new warnings. I also ran javadoc and javac-warnings ones. No new warnings there too. I am sure I didn't introduce any new release audit warnings since there are no new files in the patch. Same for @author tags. It is almost impossible to run the stand-alone test-patch command, as it exists today, on this patch.
          Hide
          Owen O'Malley added a comment -

          +1

          Show
          Owen O'Malley added a comment - +1
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Devaraj!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Devaraj!
          Owen O'Malley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/255/)
          MAPREDUCE-1421. LinuxTaskController tests failing on trunk after the commit of . Contributed by Amareshwari Sriramadasu.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #255 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/255/ ) MAPREDUCE-1421 . LinuxTaskController tests failing on trunk after the commit of . Contributed by Amareshwari Sriramadasu.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #248 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/248/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #248 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/248/ )
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          8d 13h 45m 1 Owen O'Malley 27/Jan/10 08:35
          Resolved Resolved Closed Closed
          209d 12h 44m 1 Tom White 24/Aug/10 22:20

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development