Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1383

Allow storage and caching of delegation token.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      mapreduce.job.hdfs-servers - declares hdfs servers to be used by the job, so client can pre-fetch delegation tokens for thsese servers (comma separated list of NameNodes).
    • Tags:
      security

      Description

      Client needs to obtain delegation tokens from all the NameNodes it is going to work with and pass it to the application.

      1. MAPREDUCE-1383-1.patch
        8 kB
        Boris Shkolnik
      2. MAPREDUCE-1383-2.patch
        9 kB
        Boris Shkolnik
      3. MAPREDUCE-1383-5.patch
        31 kB
        Boris Shkolnik
      4. MAPREDUCE-1383-6.patch
        31 kB
        Boris Shkolnik
      5. MAPREDUCE-1383-9.patch
        34 kB
        Boris Shkolnik
      6. MAPREDUCE-1383-10.patch
        35 kB
        Boris Shkolnik
      7. MAPREDUCE-1383-BP20-3.patch
        38 kB
        Boris Shkolnik
      8. MAPREDUCE-1383-11.patch
        34 kB
        Boris Shkolnik
      9. MAPREDUCE-1383-BP20-4.patch
        38 kB
        Boris Shkolnik
      10. MAPREDUCE-1383-BP20-5.patch
        38 kB
        Boris Shkolnik
      11. MAPREDUCE-1383-BP20-6.patch
        38 kB
        Boris Shkolnik
      12. MAPREDUCE-1383-BP20-7.patch
        48 kB
        Boris Shkolnik

        Issue Links

          Activity

          Hide
          Boris Shkolnik added a comment -

          before submitting a job - contact all the namenodes (specified in the conf) to get their corresponding delegation token.

          Show
          Boris Shkolnik added a comment - before submitting a job - contact all the namenodes (specified in the conf) to get their corresponding delegation token.
          Hide
          Boris Shkolnik added a comment -

          changed DfsClient to DistributedFileSystem
          changed references thru tokenStorage to TokenCache.

          Show
          Boris Shkolnik added a comment - changed DfsClient to DistributedFileSystem changed references thru tokenStorage to TokenCache.
          Hide
          Devaraj Das added a comment -

          After a quick look at the patch, some comments:
          1) The call to populateTokenCache should be done just before the submitJob call (so that we definitely have all the information to do with namenodes the job wishes to talk to at runtime, even those that are potentially added within calls to user's InputFormat.getSplits)
          2) The configs mapreduce.job.hdfs-servers and mapreduce.job.kerberos.jtprinicipal could probably be defined in JobContext.java.
          3) Also, though not directly related to this patch, but could you please move the JOB_TOKEN_FILE definition from JobContext.java to TokenStorage.java. Ditto for SecureShuffleUtils.JOB_TOKEN_FILENAME

          Show
          Devaraj Das added a comment - After a quick look at the patch, some comments: 1) The call to populateTokenCache should be done just before the submitJob call (so that we definitely have all the information to do with namenodes the job wishes to talk to at runtime, even those that are potentially added within calls to user's InputFormat.getSplits) 2) The configs mapreduce.job.hdfs-servers and mapreduce.job.kerberos.jtprinicipal could probably be defined in JobContext.java. 3) Also, though not directly related to this patch, but could you please move the JOB_TOKEN_FILE definition from JobContext.java to TokenStorage.java. Ditto for SecureShuffleUtils.JOB_TOKEN_FILENAME
          Hide
          Boris Shkolnik added a comment -

          preliminary patch

          Show
          Boris Shkolnik added a comment - preliminary patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431493/MAPREDUCE-1383-6.patch
          against trunk revision 903508.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431493/MAPREDUCE-1383-6.patch against trunk revision 903508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/409/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          [exec] compile-aspects:
          [exec] [echo] 1.6
          [exec] [echo] Start weaving aspects in place
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:39 [error] The import org.apache.hadoop.hdfs cannot be resolved
          [exec] [iajc] import org.apache.hadoop.hdfs.DistributedFileSystem;
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:40 [error] The import org.apache.hadoop.hdfs cannot be resolved
          [exec] [iajc] import org.apache.hadoop.hdfs.security.token.DelegationTokenIdentifier;
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:41 [error] The import org.apache.hadoop.hdfs cannot be resolved
          [exec] [iajc] import org.apache.hadoop.hdfs.server.namenode.NameNode;
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:32 [error] The import org.apache.hadoop.hdfs cannot be resolved
          [exec] [iajc] import org.apache.hadoop.hdfs.DistributedFileSystem;
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:33 [error] The import org.apache.hadoop.hdfs cannot be resolved
          [exec] [iajc] import org.apache.hadoop.hdfs.security.token.DelegationTokenIdentifier;
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:99 [error] DelegationTokenIdentifier cannot be resolved to a type
          [exec] [iajc] public static Token<DelegationTokenIdentifier> getDelegationToken(String namenode) {
          [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:181 [error] DistributedFileSystem cannot be resolved to a type
          [exec] [iajc] if(fs instanceof DistributedFileSystem) {
          [exec] [iajc] ^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:182 [error] DistributedFileSystem cannot be resolved to a type
          [exec] [iajc] DistributedFileSystem dfs = (DistributedFileSystem)fs;
          [exec] [iajc] ^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:182 [error] DistributedFileSystem cannot be resolved to a type
          [exec] [iajc] DistributedFileSystem dfs = (DistributedFileSystem)fs;
          [exec] [iajc] ^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:186 [error] DelegationTokenIdentifier cannot be resolved to a type
          [exec] [iajc] Token<DelegationTokenIdentifier> token =
          [exec] [iajc] ^^^^^^^^^^^^^^^^
          [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:187 [error] The method getDelegationToken(String) is undefined for the type TokenCache
          [exec] [iajc] TokenCache.getDelegationToken(fs_uri);
          [exec] [iajc] ^^^^^^^
          [exec] [iajc]
          [exec] [iajc] 11 errors
          [exec]
          [exec] BUILD FAILED

          This is there is the log of the build. The reason for this is the hdfs jar is not present in the classpath when compile-aspects is running. Is there a way to avoid having to import hdfs.* in the patch.

          Show
          Devaraj Das added a comment - [exec] compile-aspects: [exec] [echo] 1.6 [exec] [echo] Start weaving aspects in place [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:39 [error] The import org.apache.hadoop.hdfs cannot be resolved [exec] [iajc] import org.apache.hadoop.hdfs.DistributedFileSystem; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:40 [error] The import org.apache.hadoop.hdfs cannot be resolved [exec] [iajc] import org.apache.hadoop.hdfs.security.token.DelegationTokenIdentifier; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/JobSubmitter.java:41 [error] The import org.apache.hadoop.hdfs cannot be resolved [exec] [iajc] import org.apache.hadoop.hdfs.server.namenode.NameNode; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:32 [error] The import org.apache.hadoop.hdfs cannot be resolved [exec] [iajc] import org.apache.hadoop.hdfs.DistributedFileSystem; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:33 [error] The import org.apache.hadoop.hdfs cannot be resolved [exec] [iajc] import org.apache.hadoop.hdfs.security.token.DelegationTokenIdentifier; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:99 [error] DelegationTokenIdentifier cannot be resolved to a type [exec] [iajc] public static Token<DelegationTokenIdentifier> getDelegationToken(String namenode) { [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:181 [error] DistributedFileSystem cannot be resolved to a type [exec] [iajc] if(fs instanceof DistributedFileSystem) { [exec] [iajc] ^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:182 [error] DistributedFileSystem cannot be resolved to a type [exec] [iajc] DistributedFileSystem dfs = (DistributedFileSystem)fs; [exec] [iajc] ^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:182 [error] DistributedFileSystem cannot be resolved to a type [exec] [iajc] DistributedFileSystem dfs = (DistributedFileSystem)fs; [exec] [iajc] ^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:186 [error] DelegationTokenIdentifier cannot be resolved to a type [exec] [iajc] Token<DelegationTokenIdentifier> token = [exec] [iajc] ^^^^^^^^^^^^^^^^ [exec] [iajc] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h6.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java:187 [error] The method getDelegationToken(String) is undefined for the type TokenCache [exec] [iajc] TokenCache.getDelegationToken(fs_uri); [exec] [iajc] ^^^^^^^ [exec] [iajc] [exec] [iajc] 11 errors [exec] [exec] BUILD FAILED This is there is the log of the build. The reason for this is the hdfs jar is not present in the classpath when compile-aspects is running. Is there a way to avoid having to import hdfs.* in the patch.
          Hide
          Devaraj Das added a comment -

          Some comments:
          1) Remove the LOG.info statements from TokenCache
          2) Please add a method in TokenCache to load tokens from a file specified in the argument (rather than going indirectly through the conf). Then in Child.java, you can call that API directly rather than setting the file in conf.
          3) Does it make sense to have a TokenCache.addPathsForGettingDelegationToken(Path[]) that's called from the places where you currently have TokenCache.obtainTokensForNamenodes. Then just before job submission you make one call to TokenCache.obtainTokensForNamenodes. That way we will be sure that there is not more than one per a namenode to get delegation tokens.
          4) You define getDelegationTokens in TrackerDistributedCacheManager but don't invoke it. Also, you just need to pass the paths to the TokenCache.obtainTokensForNamenodes because that's already checking for duplicate entries.
          We discussed offline that you need to fix the build so that the hdfs jars are in the classpath for the aspects compilation..

          Show
          Devaraj Das added a comment - Some comments: 1) Remove the LOG.info statements from TokenCache 2) Please add a method in TokenCache to load tokens from a file specified in the argument (rather than going indirectly through the conf). Then in Child.java, you can call that API directly rather than setting the file in conf. 3) Does it make sense to have a TokenCache.addPathsForGettingDelegationToken(Path[]) that's called from the places where you currently have TokenCache.obtainTokensForNamenodes. Then just before job submission you make one call to TokenCache.obtainTokensForNamenodes. That way we will be sure that there is not more than one per a namenode to get delegation tokens. 4) You define getDelegationTokens in TrackerDistributedCacheManager but don't invoke it. Also, you just need to pass the paths to the TokenCache.obtainTokensForNamenodes because that's already checking for duplicate entries. We discussed offline that you need to fix the build so that the hdfs jars are in the classpath for the aspects compilation..
          Hide
          Devaraj Das added a comment -

          And yes, as per the offline discussion with Kan you need to have host:port instead of URIs for the service field of the delegation tokens.

          Show
          Devaraj Das added a comment - And yes, as per the offline discussion with Kan you need to have host:port instead of URIs for the service field of the delegation tokens.
          Hide
          Boris Shkolnik added a comment -

          addressed review comments.
          changed ivy.xml in mamuk to include hdfs.jar
          changed service field in the Delegation Token to "inet_addr:port"

          Show
          Boris Shkolnik added a comment - addressed review comments. changed ivy.xml in mamuk to include hdfs.jar changed service field in the Delegation Token to "inet_addr:port"
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431687/MAPREDUCE-1383-9.patch
          against trunk revision 903563.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431687/MAPREDUCE-1383-9.patch against trunk revision 903563. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/414/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          Two following tests are failing:
          TestTaskTrackerBlacklisting
          org.apache.hadoop.streaming.TestLoadTypedBytes

          I ran them manually and they worked fine.
          There is a jira for TestTaskTrackerBlacklisting intermittent failure (MAPREDUCE-1412).

          Show
          Boris Shkolnik added a comment - Two following tests are failing: TestTaskTrackerBlacklisting org.apache.hadoop.streaming.TestLoadTypedBytes I ran them manually and they worked fine. There is a jira for TestTaskTrackerBlacklisting intermittent failure ( MAPREDUCE-1412 ).
          Hide
          Boris Shkolnik added a comment -

          make sure DelegetionToken.setService() is "/IP:PORT"

          Show
          Boris Shkolnik added a comment - make sure DelegetionToken.setService() is "/IP:PORT"
          Hide
          Devaraj Das added a comment -

          The "/IP:PORT" looks odd. Please change the service to be <IP-ADDRESS-OF-HOST>:PORT. Other than that patch looks fine.

          Show
          Devaraj Das added a comment - The "/IP:PORT" looks odd. Please change the service to be <IP-ADDRESS-OF-HOST>:PORT. Other than that patch looks fine.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431732/MAPREDUCE-1383-10.patch
          against trunk revision 903563.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431732/MAPREDUCE-1383-10.patch against trunk revision 903563. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/415/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          Patch for earlier version of Hadoop. Not for commit here.

          Show
          Boris Shkolnik added a comment - Patch for earlier version of Hadoop. Not for commit here.
          Hide
          Boris Shkolnik added a comment -

          The "/IP:PORT" looks odd. Please change the service to be <IP-ADDRESS-OF-HOST>:PORT.

          After discussion with Kan, we aggreed on "IP:PORT". Kan will match in his implementation.

          here is my diff:
          < - sb.append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port);

          > + sb.append("/").append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port);

          Show
          Boris Shkolnik added a comment - The "/IP:PORT" looks odd. Please change the service to be <IP-ADDRESS-OF-HOST>:PORT. After discussion with Kan, we aggreed on "IP:PORT". Kan will match in his implementation. here is my diff: < - sb.append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port); — > + sb.append("/").append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port);
          Hide
          Boris Shkolnik added a comment -

          same change (ip:port) for backport.

          Show
          Boris Shkolnik added a comment - same change (ip:port) for backport.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Boris!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Boris!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)
          . Automates fetching of delegation tokens in File*Formats Distributed Cache and Distcp. Also, provides a config mapreduce.job.hdfs-servers that the jobs can populate with a comma separated list of namenodes. The job client automatically fetches delegation tokens from those namenodes. Contributed by Boris Shkolnik.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ ) . Automates fetching of delegation tokens in File*Formats Distributed Cache and Distcp. Also, provides a config mapreduce.job.hdfs-servers that the jobs can populate with a comma separated list of namenodes. The job client automatically fetches delegation tokens from those namenodes. Contributed by Boris Shkolnik.
          Hide
          Devaraj Das added a comment -

          The 0.20 patch has the TestTokenCache.java deleted. Please fix that and upload a new one. Other than that looks fine.

          Show
          Devaraj Das added a comment - The 0.20 patch has the TestTokenCache.java deleted. Please fix that and upload a new one. Other than that looks fine.
          Hide
          Boris Shkolnik added a comment -

          backport. fixed a typo (append()=>sb.append) in TokenCache.java

          Show
          Boris Shkolnik added a comment - backport. fixed a typo (append()=>sb.append) in TokenCache.java
          Hide
          Boris Shkolnik added a comment -

          merged backport

          Show
          Boris Shkolnik added a comment - merged backport
          Hide
          Boris Shkolnik added a comment -

          moved TestTokenCache.java from hadoop/security to hadoop/mapreduce/security

          Show
          Boris Shkolnik added a comment - moved TestTokenCache.java from hadoop/security to hadoop/mapreduce/security
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )

            People

            • Assignee:
              Boris Shkolnik
              Reporter:
              Boris Shkolnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development