Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3531

Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Filling this Jira a bit late
      Started 350 cluster
      sbummited large sleep job.
      Foud that job was not running as RM has not allocated resouces to it.

      2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
      2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
      type NODE_UPDATE to the scheduler
      java.lang.IllegalArgumentException: Invalid key to HMAC computation
              at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
              at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
              atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
              at java.lang.Thread.run(Thread.java:619)
      Caused by: java.security.InvalidKeyException: Secret key expected
              at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
              at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
              at javax.crypto.Mac.init(DashoA13*..)
              at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
              ... 14 more
      

      As this stack is from 30 Nov checkou line number may be different

      1. MR-3531.txt
        1 kB
        Robert Joseph Evans
      2. MR-3531.txt
        5 kB
        Robert Joseph Evans

        Activity

        Hide
        Robert Joseph Evans added a comment -

        Adding in a patch to fix the issue (Just changed HashMap to ConcurrentHashMap).

        I also added in a test, but I don't really like the test that much. It tries to brute force reproduce the problem. On my box it takes about 1.8 sec to run so it should not be too big of an issue right now.

        However, when trying to run without the fix I have seen it reproduce the problem, but I have also seen it essentially get into a living deadlock where all of the cores are pegged at 100% utilization, and sometimes only a kill -9 stops it form running. I am fine with removing the test, because I don't think it adds much value, but I am also fine with leaving it in.

        Show
        Robert Joseph Evans added a comment - Adding in a patch to fix the issue (Just changed HashMap to ConcurrentHashMap). I also added in a test, but I don't really like the test that much. It tries to brute force reproduce the problem. On my box it takes about 1.8 sec to run so it should not be too big of an issue right now. However, when trying to run without the fix I have seen it reproduce the problem, but I have also seen it essentially get into a living deadlock where all of the cores are pegged at 100% utilization, and sometimes only a kill -9 stops it form running. I am fine with removing the test, because I don't think it adds much value, but I am also fine with leaving it in.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507389/MR-3531.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507389/MR-3531.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        The change looks good to me. I'd prefer removing the test case if it can't reproduce the problem each time.

        Show
        Siddharth Seth added a comment - The change looks good to me. I'd prefer removing the test case if it can't reproduce the problem each time.
        Hide
        Siddharth Seth added a comment -

        Bobby, could you please upload another patch without the unit test.

        Show
        Siddharth Seth added a comment - Bobby, could you please upload another patch without the unit test.
        Hide
        Robert Joseph Evans added a comment -

        This patch does not have the test in it.

        Show
        Robert Joseph Evans added a comment - This patch does not have the test in it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507428/MR-3531.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507428/MR-3531.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +! for the fix as is. Atleast we could use that brute-force test to reproduce it sometimes, validating the fix is mostly the correct one.

        Show
        Vinod Kumar Vavilapalli added a comment - +! for the fix as is. Atleast we could use that brute-force test to reproduce it sometimes, validating the fix is mostly the correct one.
        Hide
        Siddharth Seth added a comment -

        Committed to trunk and branch-0.23. Thanks Bobby.

        Show
        Siddharth Seth added a comment - Committed to trunk and branch-0.23. Thanks Bobby.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1435 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1435/)
        MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1435 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1435/ ) MAPREDUCE-3531 . Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1508/)
        MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1508/ ) MAPREDUCE-3531 . Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #281 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/281/)
        merge MAPREDUCE-3531 from trunk

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #281 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/281/ ) merge MAPREDUCE-3531 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1459 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1459/)
        MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1459 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1459/ ) MAPREDUCE-3531 . Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #291 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/291/)
        merge MAPREDUCE-3531 from trunk

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #291 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/291/ ) merge MAPREDUCE-3531 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #303 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/303/)
        merge MAPREDUCE-3531 from trunk

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #303 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/303/ ) merge MAPREDUCE-3531 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/128/)
        merge MAPREDUCE-3531 from trunk

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/128/ ) merge MAPREDUCE-3531 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #108 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/)
        merge MAPREDUCE-3531 from trunk

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #108 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/ ) merge MAPREDUCE-3531 from trunk sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/)
        MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/ ) MAPREDUCE-3531 . Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development