Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3531

Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Filling this Jira a bit late
      Started 350 cluster
      sbummited large sleep job.
      Foud that job was not running as RM has not allocated resouces to it.

      2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
      2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
      type NODE_UPDATE to the scheduler
      java.lang.IllegalArgumentException: Invalid key to HMAC computation
              at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
              at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
              atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
              at java.lang.Thread.run(Thread.java:619)
      Caused by: java.security.InvalidKeyException: Secret key expected
              at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
              at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
              at javax.crypto.Mac.init(DashoA13*..)
              at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
              ... 14 more
      

      As this stack is from 30 Nov checkou line number may be different

      Attachments

        1. MR-3531.txt
          1 kB
          Robert Joseph Evans
        2. MR-3531.txt
          5 kB
          Robert Joseph Evans

        Activity

          People

            revans2 Robert Joseph Evans
            karams Karam Singh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: