Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4148

MapReduce should not have a compile-time dependency on HDFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.

      1. MAPREDUCE-4148.patch
        28 kB
        Tom White
      2. MAPREDUCE-4148.patch
        27 kB
        Tom White
      3. MAPREDUCE-4148.patch
        15 kB
        Tom White
      4. MAPREDUCE-4148.patch
        15 kB
        Tom White
      5. MAPREDUCE-4148.patch
        15 kB
        Tom White
      6. MAPREDUCE-4148.patch
        15 kB
        Tom White
      7. MAPREDUCE-4148.patch
        15 kB
        Tom White

        Activity

        Hide
        Tom White added a comment -

        This patch removes the dependency by introducing a TokenIdentifierFactory so that TokenIdentifier instances can be created from their byte array representations, without the client having to know the TokenIdentifier subtype.

        Show
        Tom White added a comment - This patch removes the dependency by introducing a TokenIdentifierFactory so that TokenIdentifier instances can be created from their byte array representations, without the client having to know the TokenIdentifier subtype.
        Hide
        Daryn Sharp added a comment -

        Very nice! I've been wanting to do something similar to allow tokens to decode their identifiers for quite awhile now. Thoughts/suggestions:

        Token#decodeIdentifier() would be really useful and prevent callers from having to know about the factory. This would very nearly abstract away the details. It could either delegate to the factory as-is, or just query the factory for the class, or maybe even Token can host the kind/class registration method. There are a number of other places in the code, ex. AbstractDelegationTokenSecretManager#renewToken(Token) that could benefit from such a method.

        I'm not too fond of AbstractDelegationTokenIdentifier#stringifyToken(Token). It creates a circular relationship. A TokenIdentifier really shouldn't have to know about a Token wrapper. How removing the inversion of knowledge, and update Token#toString() to use token.decodeIdentifier()? If the value is null because the class isn't available, it can print the raw bytes like it does now.

        TokenIdentifierFactory.createIdentifier is using ReflectionUtils.newInstance(Configuration) whose main purpose is to invoke setConf(conf) and/or MR's configure(conf) which isn't applicable in this case. How about directly using class.newInstance()?

        I think there's pitfalls with using static class inits for the factory registration. The identifier class has to be loaded (sorry if I'm stating the obvious: not just imported, but something referenced from it) before a token can be decoded. In general this probably means a token has to be created before other tokens can be decoded. Perhaps the static blocks should become a static class method that's invoked by the secret manager, since we know the secret manager is instantiated before token manipulation. Although, that limits token ident decoding only to tokens owned by the daemon, which would leave a client out of luck. Maybe you could get fancy with a class loader.

        Show
        Daryn Sharp added a comment - Very nice! I've been wanting to do something similar to allow tokens to decode their identifiers for quite awhile now. Thoughts/suggestions: Token#decodeIdentifier() would be really useful and prevent callers from having to know about the factory. This would very nearly abstract away the details. It could either delegate to the factory as-is, or just query the factory for the class, or maybe even Token can host the kind/class registration method. There are a number of other places in the code, ex. AbstractDelegationTokenSecretManager#renewToken(Token) that could benefit from such a method. I'm not too fond of AbstractDelegationTokenIdentifier#stringifyToken(Token) . It creates a circular relationship. A TokenIdentifier really shouldn't have to know about a Token wrapper. How removing the inversion of knowledge, and update Token#toString() to use token.decodeIdentifier() ? If the value is null because the class isn't available, it can print the raw bytes like it does now. TokenIdentifierFactory.createIdentifier is using ReflectionUtils.newInstance(Configuration) whose main purpose is to invoke setConf(conf) and/or MR's configure(conf) which isn't applicable in this case. How about directly using class.newInstance() ? I think there's pitfalls with using static class inits for the factory registration. The identifier class has to be loaded (sorry if I'm stating the obvious: not just imported, but something referenced from it) before a token can be decoded. In general this probably means a token has to be created before other tokens can be decoded. Perhaps the static blocks should become a static class method that's invoked by the secret manager, since we know the secret manager is instantiated before token manipulation. Although, that limits token ident decoding only to tokens owned by the daemon, which would leave a client out of luck. Maybe you could get fancy with a class loader.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 16 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 16 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 16 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 16 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        I agree with Daryn on this. I don't really like using the static block to register things. We have had issues in the past in the RM where renewing delegation tokens did not work because the HDFS configuration files were not loaded in a static block because there was no reference to the class with the static block in the code being run. It was a beast to debug too.

        I would prefer to see something that uses a ServiceLoader, or like Daryn said something from the secret manager could ensure that those static blocks are called for all of them that it cares about, but I think the service loader is the direction that a lot of the code has been going in.

        Show
        Robert Joseph Evans added a comment - I agree with Daryn on this. I don't really like using the static block to register things. We have had issues in the past in the RM where renewing delegation tokens did not work because the HDFS configuration files were not loaded in a static block because there was no reference to the class with the static block in the code being run. It was a beast to debug too. I would prefer to see something that uses a ServiceLoader, or like Daryn said something from the secret manager could ensure that those static blocks are called for all of them that it cares about, but I think the service loader is the direction that a lot of the code has been going in.
        Hide
        Tom White added a comment -

        Thanks for the reviews. I agree that using a ServiceLoader is the best approach for this one.

        Show
        Tom White added a comment - Thanks for the reviews. I agree that using a ServiceLoader is the best approach for this one.
        Hide
        Tom White added a comment -

        I've reworked the patch to use a service loader, and also to address Daryn's feedback. There is now a decodeIdentifier() method on Token, which is better than my original approach. The only thing I didn't change is the use of ReflectionUtils, since it's a convenient, centralized way to construct an object, and it handles the exceptions in a uniform way.

        (I suspect test-patch will fail due to HADOOP-8354.)

        Show
        Tom White added a comment - I've reworked the patch to use a service loader, and also to address Daryn's feedback. There is now a decodeIdentifier() method on Token, which is better than my original approach. The only thing I didn't change is the use of ReflectionUtils, since it's a convenient, centralized way to construct an object, and it handles the exceptions in a uniform way. (I suspect test-patch will fail due to HADOOP-8354 .)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525539/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525539/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//console This message is automatically generated.
        Hide
        Tom White added a comment -

        New patch to fix findbugs warnings.

        Show
        Tom White added a comment - New patch to fix findbugs warnings.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525562/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525562/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        I'd like to suggest caching the service loader lookups since there will be a fair amount of decoding.

        I'd like to see the kind and service printed before the decoded identifier. The vast majority of the time, it's those two fields that are critical to debugging. The ident will likely cause those to wrap to the next line.

        Otherwise, looks great!!

        Show
        Daryn Sharp added a comment - I'd like to suggest caching the service loader lookups since there will be a fair amount of decoding. I'd like to see the kind and service printed before the decoded identifier. The vast majority of the time, it's those two fields that are critical to debugging. The ident will likely cause those to wrap to the next line. Otherwise, looks great!!
        Hide
        Robert Joseph Evans added a comment -

        I like the patch a lot. It cleans things up which is really good. I think Daryn has a few minor comments but other then that I give it a +1. I would also like to try and pull this into branch-0.23. The patch seems to apply OK for branch-0.23, but if you do need any help with it I am happy to do it for you.

        Show
        Robert Joseph Evans added a comment - I like the patch a lot. It cleans things up which is really good. I think Daryn has a few minor comments but other then that I give it a +1. I would also like to try and pull this into branch-0.23. The patch seems to apply OK for branch-0.23, but if you do need any help with it I am happy to do it for you.
        Hide
        Tom White added a comment -

        New patch addressing Daryn's two points.

        Show
        Tom White added a comment - New patch addressing Daryn's two points.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525651/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525651/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        Just a question: is there an advantage to using google's maps? Given this jira's title, it seems ironic to add a dep.

        I've been discouraged from using static class init blocks due to the wonky exceptions it may cause. I'd consider having something like a static "getClassForIdentifier" that populates a map on first call from decodeIdentifier.

        I missed it before, but you can pass a null conf to ReflectionUtils.newInstance. It shorts out trying to set the conf when null, and it also avoids instantiating unnecessary objects. Configs also hang around in a weak hash map for awhile...

        Mainly because redundancy drives me nuts, but also because some errant exception could cause addBinaryBuffer to be called twice, would you consider this?

          private void identifierToString(StringBuilder buffer) {
            T id = null;
            try {
              id = decodeIdentifier();
            } finally {
              if (id != null) {
                buffer.append("(").append(id).append(")");
              } else {
                addBinaryBuffer(buffer, identifier);
              }
            }
          }
        
        Show
        Daryn Sharp added a comment - Just a question: is there an advantage to using google's maps? Given this jira's title, it seems ironic to add a dep. I've been discouraged from using static class init blocks due to the wonky exceptions it may cause. I'd consider having something like a static "getClassForIdentifier" that populates a map on first call from decodeIdentifier . I missed it before, but you can pass a null conf to ReflectionUtils.newInstance . It shorts out trying to set the conf when null, and it also avoids instantiating unnecessary objects. Configs also hang around in a weak hash map for awhile... Mainly because redundancy drives me nuts, but also because some errant exception could cause addBinaryBuffer to be called twice, would you consider this? private void identifierToString(StringBuilder buffer) { T id = null ; try { id = decodeIdentifier(); } finally { if (id != null ) { buffer.append( "(" ).append(id).append( ")" ); } else { addBinaryBuffer(buffer, identifier); } } }
        Hide
        Tom White added a comment -

        New patch addressing Daryn's feedback. I left in the Guava Maps usage since we already have that dependency.

        Show
        Tom White added a comment - New patch addressing Daryn's feedback. I left in the Guava Maps usage since we already have that dependency.
        Hide
        Daryn Sharp added a comment -

        +1 Great job!

        Show
        Daryn Sharp added a comment - +1 Great job!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525882/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash
        org.apache.hadoop.ipc.TestSaslRPC

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525882/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash org.apache.hadoop.ipc.TestSaslRPC +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        I like the changes, but org.apache.hadoop.ipc.TestSaslRPC appears to be a real issue, where the test is not finding the needed token identifier.

        Show
        Robert Joseph Evans added a comment - I like the changes, but org.apache.hadoop.ipc.TestSaslRPC appears to be a real issue, where the test is not finding the needed token identifier.
        Hide
        Daryn Sharp added a comment -

        Looks like a simple fix: decodeIdentifier isn't checking for a null from getClassForIdentifier when it can't find the class.

        Show
        Daryn Sharp added a comment - Looks like a simple fix: decodeIdentifier isn't checking for a null from getClassForIdentifier when it can't find the class.
        Hide
        Tom White added a comment -

        Thanks for the comments. This latest patch fixes decodeIdentifier to check for null.

        Show
        Tom White added a comment - Thanks for the comments. This latest patch fixes decodeIdentifier to check for null.
        Hide
        Daryn Sharp added a comment -

        +1 Assuming that the test cited by Bobby also works now. Great job!

        Show
        Daryn Sharp added a comment - +1 Assuming that the test cited by Bobby also works now. Great job!
        Hide
        Robert Joseph Evans added a comment -

        I am +1 for the changes too, assuming that Jenkins Comes back OK.

        Show
        Robert Joseph Evans added a comment - I am +1 for the changes too, assuming that Jenkins Comes back OK.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526294/MAPREDUCE-4148.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526294/MAPREDUCE-4148.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//console This message is automatically generated.
        Hide
        Tom White added a comment -

        I just committed this.

        Show
        Tom White added a comment - I just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2304 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2304/)
        MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2304 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2304/ ) MAPREDUCE-4148 . MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2230 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2230/)
        MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2230 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2230/ ) MAPREDUCE-4148 . MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2247/)
        MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)

        Result = ABORTED
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2247/ ) MAPREDUCE-4148 . MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/)
        MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)

        Result = FAILURE
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/ ) MAPREDUCE-4148 . MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/)
        MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/ ) MAPREDUCE-4148 . MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development