Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3083

Cannot run an MR job with HA and security enabled when second-listed NN active

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.3
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Steps to reproduce:

      • turned on ha and security
      • run a mapred job, and wait to finish
      • failover to another namenode
      • run the mapred job again, it fails.

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          Checking the job delegation token, it still indicates the original active namenode, and causes nm failed to obtain a dt for the new active nn.

          $ hdfs dfs -cat hdfs://ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens
          HDTS
               ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6
          �L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN
                                                              ha-hdfs:ns
          

          Exceptions:

          12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application application_1331619043691_0002 to ResourceManager at nn1.hadoop.local/10.177.23.38:7090
          12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job: http://nn1.hadoop.local:7050/proxy/application_1331619043691_0002/
          12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002
          12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in uber mode : false
          12/03/13 06:19:47 INFO mapreduce.Job:  map 0% reduce 0%
          12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with state FAILED due to: Application application_1331619043691_0002 failed 1 times due to AM Container for appattempt_1331619043691_0002_000001 exited with  exitCode: -1000 due to: RemoteTrace: 
          org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache
          	at org.apache.hadoop.ipc.Client.call(Client.java:1159)
          	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
          	at $Proxy28.getFileInfo(Unknown Source)
          	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
          	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
          	at $Proxy29.getFileInfo(Unknown Source)
          	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260)
          	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718)
          	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88)
          	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
          	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
          	at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
          	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
           at LocalTrace: 
          	org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache
          	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
          	at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:827)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:222)
          	at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46)
          	at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57)
          	at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355)
          	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660)
          	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654)
          
          .Failing this attempt.. Failing the application.
          12/03/13 06:19:47 INFO mapreduce.Job: Counters: 0
          Job ended: Tue Mar 13 06:19:47 UTC 2012
          The job took 3 seconds.
          
          Show
          Aaron T. Myers added a comment - Checking the job delegation token, it still indicates the original active namenode, and causes nm failed to obtain a dt for the new active nn. $ hdfs dfs -cat hdfs: //ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens HDTS ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6 �L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN ha-hdfs:ns Exceptions: 12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application application_1331619043691_0002 to ResourceManager at nn1.hadoop.local/10.177.23.38:7090 12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job: http: //nn1.hadoop.local:7050/proxy/application_1331619043691_0002/ 12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002 12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in uber mode : false 12/03/13 06:19:47 INFO mapreduce.Job: map 0% reduce 0% 12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with state FAILED due to: Application application_1331619043691_0002 failed 1 times due to AM Container for appattempt_1331619043691_0002_000001 exited with exitCode: -1000 due to: RemoteTrace: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache at org.apache.hadoop.ipc.Client.call(Client.java:1159) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188) at $Proxy28.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at $Proxy29.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49) at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157) at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) at LocalTrace: org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217) at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:827) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:222) at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46) at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654) .Failing this attempt.. Failing the application. 12/03/13 06:19:47 INFO mapreduce.Job: Counters: 0 Job ended: Tue Mar 13 06:19:47 UTC 2012 The job took 3 seconds.
          Hide
          Aaron T. Myers added a comment -

          I took a look into this this morning, and I believe I've gotten to the bottom of it.

          The trouble is not that the delegation token still identifies the first NN. In the above comment, note that you output the delegation token of the first job (_0001), which we would not expect to have changed now that you're running a new job (_0002).

          The trouble in fact seems to be that a delegation token issues by a second-listed NN, if used to communicate with the first-listed NN, will throw an exception that the client failover code doesn't interpret as warranting a failover/retry to the other NN.

          Mingjie: I believe a simpler way to reproduce this issue would just be to start up a fresh cluster, make the second-listed NN active, and then try to run a job. Can you check on that?

          The crux of the issue is that the client failover code, when initially trying to connect to the first-listed NN, will use the delegation token issued by the second-listed NN, which the first-listed NN isn't aware of. This causes a SaslException to be thrown in the o.a.h.ipc.Server, before we even get to the NN RPC code which would check an operation category and throw a StandbyException, which the client failover code knows how to deal with.

          This wasn't caught in HDFS-2904 testing because that test case gets a DT from the first-listed NN, fails over to the second-listed NN, and then ensures that the client can still communicate with the second-listed NN using the DT. However, in this scenario, both NNs are in fact aware of the DT, since the first-listed NN issued the DT, and before becoming active the second-listed NN reads all of edits logs and therefore is aware of the DT.

          One solution I can think of is to make the SecretManager in the o.a.h.ipc.Server aware of the current HA state and throw a StandbyException if the NN isn't currently active. I'm not in love with this solution, as it leaks abstractions all over the place, so would welcome alternative suggestions.

          Show
          Aaron T. Myers added a comment - I took a look into this this morning, and I believe I've gotten to the bottom of it. The trouble is not that the delegation token still identifies the first NN. In the above comment, note that you output the delegation token of the first job (_0001), which we would not expect to have changed now that you're running a new job (_0002). The trouble in fact seems to be that a delegation token issues by a second-listed NN, if used to communicate with the first-listed NN, will throw an exception that the client failover code doesn't interpret as warranting a failover/retry to the other NN. Mingjie: I believe a simpler way to reproduce this issue would just be to start up a fresh cluster, make the second-listed NN active, and then try to run a job. Can you check on that? The crux of the issue is that the client failover code, when initially trying to connect to the first-listed NN, will use the delegation token issued by the second-listed NN, which the first-listed NN isn't aware of. This causes a SaslException to be thrown in the o.a.h.ipc.Server, before we even get to the NN RPC code which would check an operation category and throw a StandbyException, which the client failover code knows how to deal with. This wasn't caught in HDFS-2904 testing because that test case gets a DT from the first-listed NN, fails over to the second-listed NN, and then ensures that the client can still communicate with the second-listed NN using the DT. However, in this scenario, both NNs are in fact aware of the DT, since the first-listed NN issued the DT, and before becoming active the second-listed NN reads all of edits logs and therefore is aware of the DT. One solution I can think of is to make the SecretManager in the o.a.h.ipc.Server aware of the current HA state and throw a StandbyException if the NN isn't currently active. I'm not in love with this solution, as it leaks abstractions all over the place, so would welcome alternative suggestions.
          Hide
          Aaron T. Myers added a comment -

          Here's a patch which addresses the issue. It includes changes in both HDFS and Common projects, so test-patch isn't going to work. I can create separate JIRAs if folks want, but I figure reviewing it would be easier as a single patch.

          No tests are included since security has to be enabled to verify the fix. To test it out, I ran the DT test script attached to HDFS-2904, with the following extra test case appended:

          # Token issued by nn2 should work when nn2 still active
          kinit -k -t ~/keytabs/$ADMIN.keytab $ADMIN/simon
          kinit -R
          hdfs haadmin -failover nn1 nn2
          rm -f /tmp/token
          hdfs fetchdt --renewer $RENEWER /tmp/token
          kdestroy
          HADOOP_TOKEN_FILE_LOCATION=/tmp/token hadoop fs -ls /
          

          All of the tests in the test script passed with this patch applied. The above test fails without the patch, and passes with it. I also successfully ran some MR jobs with the second-listed NN in the active state, and confirmed that everything worked as expected.

          Show
          Aaron T. Myers added a comment - Here's a patch which addresses the issue. It includes changes in both HDFS and Common projects, so test-patch isn't going to work. I can create separate JIRAs if folks want, but I figure reviewing it would be easier as a single patch. No tests are included since security has to be enabled to verify the fix. To test it out, I ran the DT test script attached to HDFS-2904 , with the following extra test case appended: # Token issued by nn2 should work when nn2 still active kinit -k -t ~/keytabs/$ADMIN.keytab $ADMIN/simon kinit -R hdfs haadmin -failover nn1 nn2 rm -f /tmp/token hdfs fetchdt --renewer $RENEWER /tmp/token kdestroy HADOOP_TOKEN_FILE_LOCATION=/tmp/token hadoop fs -ls / All of the tests in the test script passed with this patch applied. The above test fails without the patch, and passes with it. I also successfully ran some MR jobs with the second-listed NN in the active state, and confirmed that everything worked as expected.
          Hide
          Mingjie Lai added a comment -

          Aaron. You're right for the root cause. The order of the configured namenodes does make a difference.

          Throwing StandbyException from SecreteManager is not perfect, but okay for me.

          Good job.

          Show
          Mingjie Lai added a comment - Aaron. You're right for the root cause. The order of the configured namenodes does make a difference. Throwing StandbyException from SecreteManager is not perfect, but okay for me. Good job.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot, Mingjie. Did you perhaps get a chance to apply the patch and test out the fix?

          Show
          Aaron T. Myers added a comment - Thanks a lot, Mingjie. Did you perhaps get a chance to apply the patch and test out the fix?
          Hide
          Todd Lipcon added a comment -

          +1. This seems like the best way to fix this that I can think of as well. Can you run the HDFS tests locally to be sure before committing?

          Show
          Todd Lipcon added a comment - +1. This seems like the best way to fix this that I can think of as well. Can you run the HDFS tests locally to be sure before committing?
          Hide
          Aaron T. Myers added a comment -

          I just ran the full HDFS test suite, and they all passed.

          I'll commit this shortly based on Todd's +1. Thanks a lot for the review.

          Show
          Aaron T. Myers added a comment - I just ran the full HDFS test suite, and they all passed. I'll commit this shortly based on Todd's +1. Thanks a lot for the review.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk and branch-0.23.

          Thanks a lot for the reviews, Todd and Mingjie.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk and branch-0.23. Thanks a lot for the reviews, Todd and Mingjie.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #708 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/708/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #708 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/708/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1907 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1907 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #699 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/699/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #699 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/699/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303099) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303099 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/)
          HDFS-3083. Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/ ) HDFS-3083 . Cannot run an MR job with HA and security enabled when second-listed NN active. Contributed by Aaron T. Myers. (Revision 1303098) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1303098 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Mingjie Lai
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development