Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3990

AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Whenever node is added or removed, NodeListManager sends RMAppNodeUpdateEvent to all the applications that are in the rmcontext. But for finished/killed/failed applications it is not required to send these events. Additional check for wheather app is finished/killed/failed would minimizes the unnecessary events

        public void handle(NodesListManagerEvent event) {
          RMNode eventNode = event.getNode();
          switch (event.getType()) {
          case NODE_UNUSABLE:
            LOG.debug(eventNode + " reported unusable");
            unusableRMNodesConcurrentSet.add(eventNode);
            for(RMApp app: rmContext.getRMApps().values()) {
              this.rmContext
                  .getDispatcher()
                  .getEventHandler()
                  .handle(
                      new RMAppNodeUpdateEvent(app.getApplicationId(), eventNode,
                          RMAppNodeUpdateType.NODE_UNUSABLE));
            }
            break;
          case NODE_USABLE:
            if (unusableRMNodesConcurrentSet.contains(eventNode)) {
              LOG.debug(eventNode + " reported usable");
              unusableRMNodesConcurrentSet.remove(eventNode);
            }
            for (RMApp app : rmContext.getRMApps().values()) {
              this.rmContext
                  .getDispatcher()
                  .getEventHandler()
                  .handle(
                      new RMAppNodeUpdateEvent(app.getApplicationId(), eventNode,
                          RMAppNodeUpdateType.NODE_USABLE));
            }
            break;
          default:
            LOG.error("Ignoring invalid eventtype " + event.getType());
          }
        }
      
      1. 0001-YARN-3990.patch
        2 kB
        Bibin A Chundatt
      2. 0002-YARN-3990.patch
        8 kB
        Bibin A Chundatt
      3. 0003-YARN-3990.patch
        9 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Rohith Sharma K S

          2015-07-29 19:39:03,409 | INFO  | ResourceManager Event Processor | Added node host-7:26009 clusterResource: <memory:178400, vCores:64> | CapacityScheduler.java:1358
          2015-07-29 19:39:03,409 | INFO  | AsyncDispatcher event handler | Size of event-queue is 3000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,409 | DEBUG | Socket Reader #1 for port 26003 |  got #2125 | Server.java:1790
          2015-07-29 19:39:03,409 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER | Server.java:2058
          2015-07-29 19:39:03,410 | DEBUG | IPC Server handler 7 on 26003 | PrivilegedAction as:mapred/hadoop.hadoop.com@HADOOP.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2082) | UserGroupInformation.java:1696
          2015-07-29 19:39:03,410 | INFO  | AsyncDispatcher event handler | Size of event-queue is 4000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,410 | INFO  | AsyncDispatcher event handler | Size of event-queue is 5000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,411 | INFO  | AsyncDispatcher event handler | Size of event-queue is 6000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,412 | INFO  | AsyncDispatcher event handler | Size of event-queue is 7000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,412 | INFO  | IPC Server handler 7 on 26003 | Size of event-queue is 7000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,412 | INFO  | AsyncDispatcher event handler | Size of event-queue is 8000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,413 | INFO  | AsyncDispatcher event handler | Size of event-queue is 9000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,414 | INFO  | AsyncDispatcher event handler | Size of event-queue is 10000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,414 | INFO  | AsyncDispatcher event handler | Size of event-queue is 11000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,415 | DEBUG | IPC Server handler 7 on 26003 | Served: nodeHeartbeat queueTime= 1 procesingTime= 5 | ProtobufRpcEngine.java:631
          2015-07-29 19:39:03,415 | INFO  | AsyncDispatcher event handler | Size of event-queue is 12000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | Adding saslServer wrapped token of size 100 as call response. | Server.java:2460
          2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: responding to org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 | Server.java:994
          2015-07-29 19:39:03,416 | INFO  | AsyncDispatcher event handler | Size of event-queue is 13000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: responding to org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 Wrote 118 bytes. | Server.java:1013
          2015-07-29 19:39:03,416 | INFO  | AsyncDispatcher event handler | Size of event-queue is 14000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,417 | INFO  | AsyncDispatcher event handler | Size of event-queue is 15000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,418 | INFO  | AsyncDispatcher event handler | Size of event-queue is 16000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,419 | INFO  | AsyncDispatcher event handler | Size of event-queue is 17000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,419 | INFO  | AsyncDispatcher event handler | Size of event-queue is 18000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,420 | INFO  | AsyncDispatcher event handler | Size of event-queue is 19000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,421 | INFO  | AsyncDispatcher event handler | Size of event-queue is 20000 | AsyncDispatcher.java:235
          2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher.java:166
          2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Processing event for application_1438101193238_224125 of type NODE_UPDATE | RMAppImpl.java:741
          2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher.java:166
          2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Processing event for application_1438101193238_224126 of type NODE_UPDATE | RMAppImpl.java:741
          2015-07-29 19:39:03,422 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher.
          
          

          Was able to reproduce the same. Attaching logs

          Show
          bibinchundatt Bibin A Chundatt added a comment - Rohith Sharma K S 2015-07-29 19:39:03,409 | INFO | ResourceManager Event Processor | Added node host-7:26009 clusterResource: <memory:178400, vCores:64> | CapacityScheduler.java:1358 2015-07-29 19:39:03,409 | INFO | AsyncDispatcher event handler | Size of event-queue is 3000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,409 | DEBUG | Socket Reader #1 for port 26003 | got #2125 | Server.java:1790 2015-07-29 19:39:03,409 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER | Server.java:2058 2015-07-29 19:39:03,410 | DEBUG | IPC Server handler 7 on 26003 | PrivilegedAction as:mapred/hadoop.hadoop.com@HADOOP.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2082) | UserGroupInformation.java:1696 2015-07-29 19:39:03,410 | INFO | AsyncDispatcher event handler | Size of event-queue is 4000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,410 | INFO | AsyncDispatcher event handler | Size of event-queue is 5000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,411 | INFO | AsyncDispatcher event handler | Size of event-queue is 6000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,412 | INFO | AsyncDispatcher event handler | Size of event-queue is 7000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,412 | INFO | IPC Server handler 7 on 26003 | Size of event-queue is 7000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,412 | INFO | AsyncDispatcher event handler | Size of event-queue is 8000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,413 | INFO | AsyncDispatcher event handler | Size of event-queue is 9000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,414 | INFO | AsyncDispatcher event handler | Size of event-queue is 10000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,414 | INFO | AsyncDispatcher event handler | Size of event-queue is 11000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,415 | DEBUG | IPC Server handler 7 on 26003 | Served: nodeHeartbeat queueTime= 1 procesingTime= 5 | ProtobufRpcEngine.java:631 2015-07-29 19:39:03,415 | INFO | AsyncDispatcher event handler | Size of event-queue is 12000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | Adding saslServer wrapped token of size 100 as call response. | Server.java:2460 2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: responding to org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 | Server.java:994 2015-07-29 19:39:03,416 | INFO | AsyncDispatcher event handler | Size of event-queue is 13000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,416 | DEBUG | IPC Server handler 7 on 26003 | IPC Server handler 7 on 26003: responding to org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from 172.168.100.7:24999 Call#2125 Retry#0 Wrote 118 bytes. | Server.java:1013 2015-07-29 19:39:03,416 | INFO | AsyncDispatcher event handler | Size of event-queue is 14000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,417 | INFO | AsyncDispatcher event handler | Size of event-queue is 15000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,418 | INFO | AsyncDispatcher event handler | Size of event-queue is 16000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,419 | INFO | AsyncDispatcher event handler | Size of event-queue is 17000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,419 | INFO | AsyncDispatcher event handler | Size of event-queue is 18000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,420 | INFO | AsyncDispatcher event handler | Size of event-queue is 19000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,421 | INFO | AsyncDispatcher event handler | Size of event-queue is 20000 | AsyncDispatcher.java:235 2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher.java:166 2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Processing event for application_1438101193238_224125 of type NODE_UPDATE | RMAppImpl.java:741 2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher.java:166 2015-07-29 19:39:03,421 | DEBUG | AsyncDispatcher event handler | Processing event for application_1438101193238_224126 of type NODE_UPDATE | RMAppImpl.java:741 2015-07-29 19:39:03,422 | DEBUG | AsyncDispatcher event handler | Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppNodeUpdateEvent.EventType: NODE_UPDATE | AsyncDispatcher. Was able to reproduce the same. Attaching logs
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          thanks Bibin A Chundatt for reproducing the issue. I believe in you clustesr appsCompleted/appsRunning are 20000 and max number of completed apps to keep is set to 20k?

          Show
          rohithsharma Rohith Sharma K S added a comment - thanks Bibin A Chundatt for reproducing the issue. I believe in you clustesr appsCompleted/appsRunning are 20000 and max number of completed apps to keep is set to 20k?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Rohith Sharma K S Attaching patch for initial review

          Show
          bibinchundatt Bibin A Chundatt added a comment - Rohith Sharma K S Attaching patch for initial review
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Rohith Sharma K S
          Yes,Currently i have submitted about 50K+ apps and yarn.resourcemanager.max-completed-applications is set to 20K. yarn.resourcemanager.state-store.max-completed-applications default =yarn.resourcemanager.max-completed-applications

          Show
          bibinchundatt Bibin A Chundatt added a comment - Rohith Sharma K S Yes,Currently i have submitted about 50K+ apps and yarn.resourcemanager.max-completed-applications is set to 20K. yarn.resourcemanager.state-store.max-completed-applications default = yarn.resourcemanager.max-completed-applications
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 48s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 18s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 52m 1s Tests passed in hadoop-yarn-server-resourcemanager.
              89m 55s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747731/0001-YARN-3990.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6374ee0
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8705/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8705/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8705/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 18s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 52m 1s Tests passed in hadoop-yarn-server-resourcemanager.     89m 55s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747731/0001-YARN-3990.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6374ee0 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8705/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8705/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8705/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Nice catch, Rohith!

          +1 lgtm. Will commit later today to give others a chance to comment.

          Show
          jlowe Jason Lowe added a comment - Nice catch, Rohith! +1 lgtm. Will commit later today to give others a chance to comment.
          Hide
          zxu zhihai xu added a comment -

          Yes, that is a good catch!
          rmContext.getRMApps() stores both completed and running APPs, Current default value for max-completed-applications is 10000, we may save up-to 10000 NodeUpdateEvent.
          +1 too.

          Show
          zxu zhihai xu added a comment - Yes, that is a good catch! rmContext.getRMApps() stores both completed and running APPs, Current default value for max-completed-applications is 10000, we may save up-to 10000 NodeUpdateEvent. +1 too.
          Hide
          adhoot Anubhav Dhoot added a comment -

          Change looks good. It would be good to have a unit test to catch regressions.

          Show
          adhoot Anubhav Dhoot added a comment - Change looks good. It would be good to have a unit test to catch regressions.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Anubhav Dhoot . Was tricky. Attaching patch including testcase.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Anubhav Dhoot . Was tricky. Attaching patch including testcase.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 37s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 52m 38s Tests passed in hadoop-yarn-server-resourcemanager.
              90m 31s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747961/0002-YARN-3990.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ddc867ce
          javac https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/diffJavacWarnings.txt
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8714/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8714/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 37s The applied patch generated 1 additional warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 52m 38s Tests passed in hadoop-yarn-server-resourcemanager.     90m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747961/0002-YARN-3990.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ddc867ce javac https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/diffJavacWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8714/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8714/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8714/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Please look into the java and release audit warnings.

          The new TestNMUpdateEvent file probably should just be named TestNodesListManager so we can add more tests specific to the NodesListManager there.

          Rather than using mock RM and NM objects and running some RMApps it seems like we could have just isolated the NodesListManager more directly. We can hand it a mocked RMContext that returns a pre-baked list of apps that are alive and apps that are finished and then verify when we send a node usable/unusable event that the appropriate update events are sent to the appropriate apps by registering our own event handler with something like a drain or inline dispatcher. Not a must fix, just wondering if that approach was considered.

          Show
          jlowe Jason Lowe added a comment - Please look into the java and release audit warnings. The new TestNMUpdateEvent file probably should just be named TestNodesListManager so we can add more tests specific to the NodesListManager there. Rather than using mock RM and NM objects and running some RMApps it seems like we could have just isolated the NodesListManager more directly. We can hand it a mocked RMContext that returns a pre-baked list of apps that are alive and apps that are finished and then verify when we send a node usable/unusable event that the appropriate update events are sent to the appropriate apps by registering our own event handler with something like a drain or inline dispatcher. Not a must fix, just wondering if that approach was considered.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Jason Lowe
          Thank you for your comments

          java and release audit warnings.

          Done

          The new TestNMUpdateEvent file probably should just be named TestNodesListManager

          Done

          Show
          bibinchundatt Bibin A Chundatt added a comment - Jason Lowe Thank you for your comments java and release audit warnings. Done The new TestNMUpdateEvent file probably should just be named TestNodesListManager Done
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 52m 51s Tests passed in hadoop-yarn-server-resourcemanager.
              91m 32s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748025/0003-YARN-3990.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8acb30b
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8718/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8718/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8718/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 52m 51s Tests passed in hadoop-yarn-server-resourcemanager.     91m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748025/0003-YARN-3990.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8acb30b hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8718/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8718/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8718/console This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Bibin for the contribution and to zhihai and Anubhav for additional review! I committed this to trunk, branch-2, and branch-2.7.

          Show
          jlowe Jason Lowe added a comment - Thanks to Bibin for the contribution and to zhihai and Anubhav for additional review! I committed this to trunk, branch-2, and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8251/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8251/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1004 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1004/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1004 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1004/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/274/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/274/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2220 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2220/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2220 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2220/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2201/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2201/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/263/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/263/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #271 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/271/)
          YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #271 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/271/ ) YARN-3990 . AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt (jlowe: rev 32e490b6c035487e99df30ce80366446fe09bd6c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/CHANGES.txt
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1. Ran compilation and TestNodesListManager before the push. Patch applied cleanly.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. Ran compilation and TestNodesListManager before the push. Patch applied cleanly.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development