Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3030

RM is not processing heartbeat and continuously giving the message 'Node not found rebooting'

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: mrv2, resourcemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Node Manager Logs
      2011-09-19 13:39:29,816 INFO  webapp.WebApps (WebApps.java:start(162)) - Registered webapp guice modules
      2011-09-19 13:39:29,817 INFO  service.AbstractService (AbstractService.java:start(61)) - Service:org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer is started.
      2011-09-19 13:39:29,818 INFO  service.AbstractService (AbstractService.java:start(61)) - Service:Dispatcher is started.
      2011-09-19 13:39:29,819 INFO  nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:start(133)) - Configured ContainerManager Address is 10.18.52.124:45454
      2011-09-19 13:39:29,819 INFO  ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
      2011-09-19 13:39:29,822 INFO  ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(49)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
      2011-09-19 13:39:29,862 INFO  nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(165)) - Connected to ResourceManager at 0.0.0.0:8025
      2011-09-19 13:39:30,369 INFO  nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:registerWithRM(189)) - Registered with ResourceManager as 10.18.52.124:45454 with total resource of memory: 8192, 
      2011-09-19 13:39:30,369 INFO  service.AbstractService (AbstractService.java:start(61)) - Service:org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl is started.
      2011-09-19 13:39:30,371 INFO  service.AbstractService (AbstractService.java:start(61)) - Service:org.apache.hadoop.yarn.server.nodemanager.NodeManager is started.
      
      Resource Manager Logs
      2011-09-19 14:01:03,238 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      Call: protocol=org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface, method=nodeHeartbeat
      2011-09-19 14:01:04,240 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      Call: protocol=org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface, method=nodeHeartbeat
      2011-09-19 14:01:05,242 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      Call: protocol=org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface, method=nodeHeartbeat
      2011-09-19 14:01:06,244 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      Call: protocol=org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface, method=nodeHeartbeat
      2011-09-19 14:01:07,246 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      Call: protocol=org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface, method=nodeHeartbeat
      2011-09-19 14:01:08,247 INFO  resourcemanager.ResourceTrackerService (ResourceTrackerService.java:nodeHeartbeat(201)) - Node not found rebooting 10.18.52.124:45454
      

      Node Manager is registered with Resource manager and the for every heartbeat, it is printing the above message.

        Issue Links

          Activity

          Hide
          Devaraj K added a comment -

          This issue is coming due to the changes of MAPREDUCE-2965.

          NodeId.java
          public boolean equals(Object obj) {
              if (this == obj)
                return true;
              if (!super.equals(obj))
                return false;
              if (getClass() != obj.getClass())
                return false;
              NodeId other = (NodeId) obj;
              if (!this.getHost().equals(other.getHost()))
                return false;
              if (this.getPort() != other.getPort())
                return false;
              return true;
            }
          

          Here the super.equals(obj) is always returning false, nodeId are becoming not equal even if the states are equal.

          RMNode rmNode = this.rmContext.getRMNodes().get(nodeId) in ResourceTrackerService.java is always returning null, even if the node was registered already and it is causing to log continuously in resource manager logs. Due to this submitted application progress is always 0.

          Show
          Devaraj K added a comment - This issue is coming due to the changes of MAPREDUCE-2965 . NodeId.java public boolean equals( Object obj) { if ( this == obj) return true ; if (! super .equals(obj)) return false ; if (getClass() != obj.getClass()) return false ; NodeId other = (NodeId) obj; if (! this .getHost().equals(other.getHost())) return false ; if ( this .getPort() != other.getPort()) return false ; return true ; } Here the super.equals(obj) is always returning false, nodeId are becoming not equal even if the states are equal. RMNode rmNode = this.rmContext.getRMNodes().get(nodeId) in ResourceTrackerService.java is always returning null, even if the node was registered already and it is causing to log continuously in resource manager logs. Due to this submitted application progress is always 0.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Oh dear. Blame this on sloppy review.

          Devaraj (K), can you quickly provide a patch please? Do add a test inline with TestContainerId, I think the absence of the test is a culprit too.

          Thanks!

          Show
          Vinod Kumar Vavilapalli added a comment - Oh dear. Blame this on sloppy review. Devaraj (K), can you quickly provide a patch please? Do add a test inline with TestContainerId, I think the absence of the test is a culprit too. Thanks!
          Hide
          Devaraj K added a comment -

          I will upload the patch now as soon as possible.

          Show
          Devaraj K added a comment - I will upload the patch now as soon as possible.
          Hide
          Devaraj K added a comment -

          Updated patch with the testcase.

          Show
          Devaraj K added a comment - Updated patch with the testcase.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12495062/MAPREDUCE-3030.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/780//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/780//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495062/MAPREDUCE-3030.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/780//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/780//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the straightforward patch.

          Just committed this to trunk and branch 0.23 so that no one gets blocked on this.

          Thanks Devaraj (K)!

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the straightforward patch. Just committed this to trunk and branch 0.23 so that no one gets blocked on this. Thanks Devaraj (K)!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #912 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/912/)
          MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #912 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/912/ ) MAPREDUCE-3030 . Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #989 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/989/)
          MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #989 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/989/ ) MAPREDUCE-3030 . Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #929 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/929/)
          MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #929 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/929/ ) MAPREDUCE-3030 . Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/)
          MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/ ) MAPREDUCE-3030 . Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/836/)
          MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/836/ ) MAPREDUCE-3030 . Fixed a bug in NodeId.equals() that was causing RM to reject all NMs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172638 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/12/)
          MAPREDUCE-3030. svn merge -c r1172638 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172639
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/12/ ) MAPREDUCE-3030 . svn merge -c r1172638 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172639 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/)
          MAPREDUCE-3030. svn merge -c r1172638 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172639
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/ ) MAPREDUCE-3030 . svn merge -c r1172638 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172639 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestNodeId.java

            People

            • Assignee:
              Devaraj K
              Reporter:
              Devaraj K
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development