Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3530

Sometimes NODE_UPDATE to the scheduler throws an NPE causing the scheduling to stop

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed an NPE occuring during scheduling in the ResourceManager.

      Description

      Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop but ResourceManager keeps on running.
      I have been observing intermitently for last 3 weeks.
      But with latest svn code. I tried to run sort twice and both times Job got stuck due to NPE.

      java.lang.NullPointerException
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerLaunchedOnNode(SchedulerApp.java:181)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.containerLaunchedOnNode(CapacityScheduler.java:596)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:539)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:617)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
              at java.lang.Thread.run(Thread.java:619)
      
      1. MAPREDUCE-3530.patch
        2 kB
        Arun C Murthy
      2. MAPREDUCE-3530.patch
        12 kB
        Arun C Murthy
      3. MAPREDUCE-3530-20111214.txt
        9 kB
        Vinod Kumar Vavilapalli

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        18h 25m 2 Vinod Kumar Vavilapalli 14/Dec/11 19:17
        Open Open Patch Available Patch Available
        1d 19h 16m 3 Vinod Kumar Vavilapalli 14/Dec/11 19:18
        Patch Available Patch Available Resolved Resolved
        2h 5m 1 Vinod Kumar Vavilapalli 14/Dec/11 21:24
        Resolved Resolved Closed Closed
        81d 5h 25m 1 Arun C Murthy 05/Mar/12 02:49
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #108 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.
        svn merge -c 1214476 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #108 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. svn merge -c 1214476 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #126 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/126/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.
        svn merge -c 1214476 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #126 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/126/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. svn merge -c 1214476 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #279 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/279/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.
        svn merge -c 1214476 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #279 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/279/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. svn merge -c 1214476 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1433 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1433/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1433 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1433/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1506/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1506/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #289 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/289/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.
        svn merge -c 1214476 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #289 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/289/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. svn merge -c 1214476 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1456 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1456/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1456 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1456/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214476 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #301 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/301/)
        MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy.
        svn merge -c 1214476 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #301 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/301/ ) MAPREDUCE-3530 . Fixed an NPE occuring during scheduling in the ResourceManager. Contributed by Arun C Murthy. svn merge -c 1214476 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214478 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note Fixed an NPE occuring during scheduling in the ResourceManager.
        Resolution Fixed [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Just committed this to trunk and branch-0.23. Thanks Arun!

        Show
        Vinod Kumar Vavilapalli added a comment - Just committed this to trunk and branch-0.23. Thanks Arun!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507398/MAPREDUCE-3530-20111214.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1451//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1451//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507398/MAPREDUCE-3530-20111214.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1451//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1451//console This message is automatically generated.
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.23.1 [ 12318883 ]
        Vinod Kumar Vavilapalli made changes -
        Attachment MAPREDUCE-3530-20111214.txt [ 12507398 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Updated patch with the trivial edits.

        Show
        Vinod Kumar Vavilapalli added a comment - Updated patch with the trivial edits.
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Patch looks good overall.

        Minor edits:

        • RNNodeImpl.getStateMachine() isn't used anywhere.
        • The real ContainerID etc are as easy to be created using BuilderUtils.

        I'll upload an updated patch with these trivial edits and push this in after Jenkins blesses it.

        Show
        Vinod Kumar Vavilapalli added a comment - Patch looks good overall. Minor edits: RNNodeImpl.getStateMachine() isn't used anywhere. The real ContainerID etc are as easy to be created using BuilderUtils. I'll upload an updated patch with these trivial edits and push this in after Jenkins blesses it.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507322/MAPREDUCE-3530.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1444//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1444//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507322/MAPREDUCE-3530.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1444//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1444//console This message is automatically generated.
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3530.patch [ 12507322 ]
        Hide
        Arun C Murthy added a comment -

        Same patch, alongwith a unit test.

        Show
        Arun C Murthy added a comment - Same patch, alongwith a unit test.
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        The RM should probably exit if the scheduler thread sees exceptions, instead of the RM continuing to run without the scheduler thread.

        Let's do that separately. We need this kind of checking for all components.

        Show
        Vinod Kumar Vavilapalli added a comment - The RM should probably exit if the scheduler thread sees exceptions, instead of the RM continuing to run without the scheduler thread. Let's do that separately. We need this kind of checking for all components.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507272/MAPREDUCE-3530.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1440//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1440//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507272/MAPREDUCE-3530.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1440//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1440//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        The RM should probably exit if the scheduler thread sees exceptions, instead of the RM continuing to run without the scheduler thread. That could be changed here or in a separate jira ?

        Show
        Siddharth Seth added a comment - The RM should probably exit if the scheduler thread sees exceptions, instead of the RM continuing to run without the scheduler thread. That could be changed here or in a separate jira ?
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 0.23.1 [ 12318883 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-3530.patch [ 12507272 ]
        Hide
        Arun C Murthy added a comment -

        With Vinod's help, fixed a corner case where expired/released containers aren't ignored when NM reports back (possibly late).

        I'm adding a unit test.

        Show
        Arun C Murthy added a comment - With Vinod's help, fixed a corner case where expired/released containers aren't ignored when NM reports back (possibly late). I'm adding a unit test.
        Vinod Kumar Vavilapalli made changes -
        Summary Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop Sometimes NODE_UPDATE to the scheduler throws an NPE causing the scheduling to stop
        Hide
        Vinod Kumar Vavilapalli added a comment -

        While I was testing with Karam, I worked around it by putting in a null check. We have to come up with a scenario, I couldn't find much from the logs too at that time.

        Show
        Vinod Kumar Vavilapalli added a comment - While I was testing with Karam, I worked around it by putting in a null check. We have to come up with a scenario, I couldn't find much from the logs too at that time.
        Arun C Murthy made changes -
        Assignee Vinod Kumar Vavilapalli [ vinodkv ] Arun C Murthy [ acmurthy ]
        Hide
        Arun C Murthy added a comment -

        I'll take a look.

        Show
        Arun C Murthy added a comment - I'll take a look.
        Arun C Murthy made changes -
        Assignee Vinod Kumar Vavilapalli [ vinodkv ]
        Karam Singh made changes -
        Affects Version/s 0.23.1 [ 12318883 ]
        Hide
        Karam Singh added a comment -

        Vinod was aware of it and was working on it

        Show
        Karam Singh added a comment - Vinod was aware of it and was working on it
        Mahadev konar made changes -
        Field Original Value New Value
        Priority Major [ 3 ] Blocker [ 1 ]
        Karam Singh created issue -

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development