[MAPREDUCE-3531] Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.1
• Fix Version/s: 0.23.1
• Component/s: mrv2, resourcemanager, scheduler
• Target Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Assignee: Robert Joseph Evans
• Reporter: Karam Singh
• Created: 12/Dec/11 05:42
• Updated: 05/Mar/12 02:49
• Resolved: 15/Dec/11 02:13

Description

Filling this Jira a bit late
Started 350 cluster
sbummited large sleep job.
Foud that job was not running as RM has not allocated resouces to it.

2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
type NODE_UPDATE to the scheduler
java.lang.IllegalArgumentException: Invalid key to HMAC computation
        at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
        at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
        atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.security.InvalidKeyException: Secret key expected
        at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
        at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
        at javax.crypto.Mac.init(DashoA13*..)
        at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
        ... 14 more

As this stack is from 30 Nov checkou line number may be different