Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-905

Make changes to HDFS for the new UserGroupInformation APIs (HADOOP-6299)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is about moving the HDFS code to use the new UserGroupInformation API as described in HADOOP-6299.

      1. HDFS-905.patch
        134 kB
        Jakob Homan
      2. HDFS-905.patch
        134 kB
        Jakob Homan
      3. HDFS-905-mark3.patch
        135 kB
        Jakob Homan

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Jakob!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Jakob!
          Hide
          Owen O'Malley added a comment -

          +1

          Show
          Owen O'Malley added a comment - +1
          Hide
          Jakob Homan added a comment -

          Attaching final patch. Passes all tests.
          Modified test-patch to use new common jar:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 76 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          Show
          Jakob Homan added a comment - Attaching final patch. Passes all tests. Modified test-patch to use new common jar: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 76 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431187/HDFS-905.patch
          against trunk revision 902330.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 76 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431187/HDFS-905.patch against trunk revision 902330. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 76 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/201/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          submitting patch, but will still bomb...

          Show
          Jakob Homan added a comment - submitting patch, but will still bomb...
          Hide
          Jakob Homan added a comment -

          Updated patch: addresses Owen's comments, adds datanode/namenode logging on during beginning, has raid removed, general cleanup.

          Show
          Jakob Homan added a comment - Updated patch: addresses Owen's comments, adds datanode/namenode logging on during beginning, has raid removed, general cleanup.
          Hide
          Owen O'Malley added a comment -

          This is looking good. I notice that you have some spurious whitespace changes and a fair number of lines that go past 80 characters.

          In FileChecksumServlets.java, you have:

          +        final NameNode namenode = getUGI(request).doAs(new PrivilegedExceptionA
          ction<NameNode>() {
          +          @Override
          +          public NameNode run() throws IOException {
          +            return (NameNode)context.getAttribute("name.node");
          +          }
          +        });
          

          but the context.getAttribute doesn't use the current user, it just pulls an object out of a map setup when the jetty server started.

          Show
          Owen O'Malley added a comment - This is looking good. I notice that you have some spurious whitespace changes and a fair number of lines that go past 80 characters. In FileChecksumServlets.java, you have: + final NameNode namenode = getUGI(request).doAs(new PrivilegedExceptionA ction<NameNode>() { + @Override + public NameNode run() throws IOException { + return (NameNode)context.getAttribute("name.node"); + } + }); but the context.getAttribute doesn't use the current user, it just pulls an object out of a map setup when the jetty server started.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431092/HDFS-905.patch
          against trunk revision 901316.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 76 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431092/HDFS-905.patch against trunk revision 901316. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 76 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/198/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          submitting patch, but won't compile without updated common jar...

          Show
          Jakob Homan added a comment - submitting patch, but won't compile without updated common jar...
          Hide
          Jakob Homan added a comment -

          Patch for review. Blocked by HDFS-902. Can't get the hdfsproxy cactus-based unit tests to run on either trunk or branch (times out when starting container; will investigate).

          Marking PA in case findbugs and javac warnings work, but will bomb out spectacularly on unit tests since HADOOP-6299 is not in yet. Will run and post those locally with patched common jar.

          Show
          Jakob Homan added a comment - Patch for review. Blocked by HDFS-902 . Can't get the hdfsproxy cactus-based unit tests to run on either trunk or branch (times out when starting container; will investigate). Marking PA in case findbugs and javac warnings work, but will bomb out spectacularly on unit tests since HADOOP-6299 is not in yet. Will run and post those locally with patched common jar.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development