[SLIDER-646] AgentLaunchFailureIT test failing at times - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: Slider 0.60, Slider 0.80
Component/s: test
Labels:
None

Sprint:
Slider November #1

Description

Chaos Monkey initial delay should be deterministic. It is currently set to 60 seconds. Subsequent interval is also set to 60 secs. However AgentLaunchFailureIT fails at times because the AM does not get sufficient time to startup. In one failure scenario it has been seen to fail within 300 ms of Chaos Monkey setup. This test fails about once in every 10 attempts.

Here is the test output -

------------------------------------------------------------------------------
Test set: org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 43.503 sec <<< FAILURE! - in org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
testAgentLaunchFailure(org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT)  Time elapsed: 40.903 sec  <<< FAILURE!
java.lang.AssertionError: Application Launch Failure, exit code  68
Chaos monkey triggered launch failure
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.slider.funtest.framework.CommandTestBase.createTemplatedSliderApplication(CommandTestBase.groovy:676)
        at org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT.testAgentLaunchFailure(AgentLaunchFailureIT.groovy:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Here is the AM log snippet -

2014-11-12 09:29:34,989 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(764)) - Token YARN_AM_RM_TOKEN
2014-11-12 09:29:34,990 [main] INFO  agent.AgentUtils (AgentUtils.java:getApplicationMetainfo(43)) - Reading metainfo at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,014 [main] INFO  tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading metainfo.xml of size 1995
2014-11-12 09:29:35,096 [main] INFO  agent.AgentUtils (AgentUtils.java:getDefaultConfig(64)) - Reading default config file configuration/cl-site.xml at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,102 [main] INFO  tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading configuration/cl-site.xml of size 1270
2014-11-12 09:29:35,106 [main] INFO  agent.HeartbeatMonitor (HeartbeatMonitor.java:start(46)) - Starting heartbeat monitor with interval 60000
2014-11-12 09:29:35,107 [Thread-36] DEBUG agent.HeartbeatMonitor (HeartbeatMonitor.java:run(65)) - Putting monitor to sleep for 60000 milliseconds
2014-11-12 09:29:35,181 [main] INFO  state.AppState (AppState.java:buildInstance(502)) - Adding role COMMAND_LOGGER
2014-11-12 09:29:35,181 [main] INFO  state.AppState (AppState.java:createDynamicProviderRole(585)) - Role COMMAND_LOGGER assigned priority 1
2014-11-12 09:29:35,181 [main] INFO  state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
2014-11-12 09:29:35,253 [main] DEBUG state.RoleHistory (RoleHistory.java:onBootstrap(370)) - Role history bootstrapped
2014-11-12 09:29:35,268 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2183)) - Adding Chaos Monkey scheduled every 60 seconds (0 hours -delay 60
2014-11-12 09:29:35,269 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2195)) - Chaos Monkey has triggered AM Launch failure
2014-11-12 09:29:35,269 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing stop:  exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action AM killer not enabled
2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action Container killer not enabled
2014-11-12 09:29:35,270 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2222)) - Chaos monkey not started
2014-11-12 09:29:35,271 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(879)) - HADOOP_USER_NAME='yarn'
2014-11-12 09:29:35,287 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(882)) - Registry service username =yarn
2014-11-12 09:29:35,308 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/yarn-site.xml
2014-11-12 09:29:35,342 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/core-site.xml
2014-11-12 09:29:35,375 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/hdfs-site.xml
2014-11-12 09:29:35,437 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1083)) - Service Record 
ServiceRecord{description='Slider Application Master'; external endpoints: {{
  "api" : "http://",
  "addressType" : "uri",
  "protocolType" : "webui",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091"
  } ]
}; {
  "api" : "classpath:org.apache.slider.management",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
  } ]
}; {
  "api" : "classpath:org.apache.slider.registry",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher.configurations",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher.exports",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
  } ]
}; }; internal endpoints: {{
  "api" : "classpath:org.apache.slider.agents.secure",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
  } ]
}; {
  "api" : "classpath:org.apache.slider.agents.oneway",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
  } ]
}; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
2014-11-12 09:29:35,473 [main] INFO  zk.RegistryOperationsService (RegistryOperationsService.java:bind(110)) - Bound at /users/yarn/services/org-apache-slider/test-agent-launchfail : ServiceRecord{description='Slider Application Master'; external endpoints: {{
  "api" : "http://",
  "addressType" : "uri",
  "protocolType" : "webui",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091"
  } ]
}; {
  "api" : "classpath:org.apache.slider.management",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
  } ]
}; {
  "api" : "classpath:org.apache.slider.registry",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher.configurations",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
  } ]
}; {
  "api" : "classpath:org.apache.slider.publisher.exports",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
  } ]
}; }; internal endpoints: {{
  "api" : "classpath:org.apache.slider.agents.secure",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
  } ]
}; {
  "api" : "classpath:org.apache.slider.agents.oneway",
  "addressType" : "uri",
  "protocolType" : "REST",
  "addresses" : [ {
    "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
  } ]
}; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
2014-11-12 09:29:35,535 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1085)) - Registered service under /users/yarn/services/org-apache-slider/test-agent-launchfail; absolute path /registry/users/yarn/services/org-apache-slider/test-agent-launchfail
2014-11-12 09:29:35,543 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(888)) - RM Webapp address 172.31.11.99:8088
2014-11-12 09:29:35,543 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(889)) - slider Webapp address http://172.31.11.97:37091
2014-11-12 09:29:35,543 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(892)) - Application Master Initialization Completed
2014-11-12 09:29:35,543 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:startQueueProcessing(463)) - Queue Processing started
2014-11-12 09:29:35,544 [AmExecutor-005] INFO  actions.QueueService (QueueService.java:run(171)) - QueueService processor started
2014-11-12 09:29:35,545 [AmExecutor-006] INFO  actions.QueueExecutor (QueueExecutor.java:run(68)) - Queue Executor run() started
2014-11-12 09:29:35,545 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing stop:  exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,545 [AmExecutor-006] INFO  appmaster.SliderAppMaster (ActionStopSlider.java:execute(118)) - SliderAppMasterApi.stopCluster: Chaos monkey triggered launch failure
2014-11-12 09:29:35,619 [main] DEBUG agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(117)) - Validating conf {,
"internal": {
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : {
    "create.hadoop.deployed.info" : "(no branch) @ae493e14a5f8a78bd6227e6d377bbef6",
    "create.application.build.info" : "Slider Core-0.60.0.2.2.0.0-1947 Built against commit# ${buildNumber} on Java 1.7.0_67 by yarn",
    "create.hadoop.build.info" : "2.6.0.2.2.0.0-1947",
    "create.time.millis" : "1415784562671",
    "create.time" : "12 Nov 2014 09:29:22 GMT"
  },
  "global" : {
    "internal.generated.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/generated",
    "application.name" : "test-agent-launchfail",
    "slider.cluster.directory.permissions" : "0770",
    "internal.provider.name" : "agent",
    "internal.data.dir.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/database",
    "internal.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp",
    "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
    "internal.snapshot.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/snapshot",
    "internal.chaos.monkey.interval.seconds" : "60",
    "slider.data.directory.permissions" : "0770",
    "internal.container.failure.shortlife" : "60000",
    "internal.chaos.monkey.enabled" : "true",
    "internal.am.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp/appmaster",
    "internal.container.failure.threshold" : "5"
  },
  "credentials" : { },
  "components" : { }
},
"resources": {
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : { },
  "global" : { },
  "credentials" : { },
  "components" : {
    "slider-appmaster" : {
      "yarn.memory" : "256",
      "yarn.vcores" : "1",
      "yarn.component.instances" : "1"
    },
    "COMMAND_LOGGER" : {
      "yarn.memory" : "128",
      "yarn.role.priority" : "1",
      "yarn.component.instances" : "0"
    }
  }
},
"appConf" :{
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : { },
  "global" : {
    "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
    "site.fs.default.name" : "hdfs://172.31.11.99:8020",
    "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
    "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
    "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
    "java_home" : "/usr/jdk64/jdk1.7.0_67",
    "site.global.application_id" : "CommandLogger",
    "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
    "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
    "env.MALLOC_ARENA_MAX" : "4",
    "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
    "internal.chaos.monkey.interval.seconds" : "60",
    "internal.chaos.monkey.enabled" : "true",
    "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
    "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
    "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
    "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
    "site.global.security_enabled" : "false"
  },
  "credentials" : { },
  "components" : {
    "slider-appmaster" : {
      "jvm.heapsize" : "256M",
      "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
      "site.fs.default.name" : "hdfs://172.31.11.99:8020",
      "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
      "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
      "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
      "java_home" : "/usr/jdk64/jdk1.7.0_67",
      "site.global.application_id" : "CommandLogger",
      "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
      "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
      "env.MALLOC_ARENA_MAX" : "4",
      "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
      "internal.chaos.monkey.interval.seconds" : "60",
      "internal.chaos.monkey.enabled" : "true",
      "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
      "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
      "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
      "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
      "site.global.security_enabled" : "false"
    },
    "COMMAND_LOGGER" : {
      "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
      "site.fs.default.name" : "hdfs://172.31.11.99:8020",
      "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
      "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
      "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
      "java_home" : "/usr/jdk64/jdk1.7.0_67",
      "site.global.application_id" : "CommandLogger",
      "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
      "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
      "env.MALLOC_ARENA_MAX" : "4",
      "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
      "internal.chaos.monkey.interval.seconds" : "60",
      "internal.chaos.monkey.enabled" : "true",
      "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
      "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
      "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
      "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
      "site.global.security_enabled" : "false"
    }
  }
}}
2014-11-12 09:29:35,619 [main] INFO  agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(133)) - Validating app definition .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,620 [main] DEBUG state.AppState (AppState.java:updateResourceDefinitions(649)) - Updating resources to {
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : { },
  "global" : { },
  "credentials" : { },
  "components" : {
    "slider-appmaster" : {
      "yarn.memory" : "256",
      "yarn.vcores" : "1",
      "yarn.component.instances" : "1"
    },
    "COMMAND_LOGGER" : {
      "yarn.memory" : "128",
      "yarn.role.priority" : "1",
      "yarn.component.instances" : "0"
    }
  }
}
2014-11-12 09:29:35,704 [main] INFO  state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of slider-appmaster; was 0
2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of COMMAND_LOGGER; was 0
2014-11-12 09:29:35,704 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:reviewRequestAndReleaseNodes(1524)) - reviewRequestAndReleaseNodes(flexCluster)
2014-11-12 09:29:35,705 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,705 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:waitForAMCompletionSignal(1243)) - blocking until signalled to terminate
2014-11-12 09:29:35,705 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed stop:  exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,705 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:finish(1288)) - Triggering shutdown of the AM: stop:  exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,706 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,706 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:stateChanged(1963)) - Process has exited with exit code 0 mapped to 0 -ignoring
2014-11-12 09:29:35,706 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:finish(1299)) - Stopped forked process: exit code=0
2014-11-12 09:29:35,707 [main] INFO  workflow.WorkflowCompositeService (WorkflowCompositeService.java:stateChanged(123)) - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED
2014-11-12 09:29:35,707 [main] INFO  state.AppState (AppState.java:releaseAllContainers(1843)) - Releasing 0 containers
2014-11-12 09:29:35,707 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:finish(1317)) - Application completed. Signalling finish to RM
2014-11-12 09:29:35,707 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:finish(1320)) - Unregistering AM status=FAILED message=Chaos monkey triggered launch failure
2014-11-12 09:29:35,716 [main] INFO  impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(383)) - Waiting for application to be successfully unregistered.
2014-11-12 09:29:35,818 [main] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:runService(529)) - Exiting AM; final exit code = 0
2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1559)) - in executeNodeReview(flexCluster)
2014-11-12 09:29:35,818 [main] DEBUG main.ServiceLauncher (ServiceLauncher.java:launchService(189)) - Service exited with exit code 0
2014-11-12 09:29:35,818 [AmExecutor-006] INFO  appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1561)) - Ignoring node review operation: shutdown in progress
2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG state.AppState (AppState.java:reviewRequestAndReleaseNodes(1599)) - in reviewRequestAndReleaseNodes()
2014-11-12 09:29:35,818 [AmExecutor-006] INFO  state.AppState (AppState.java:reviewOneRole(1684)) - Reviewing RoleStatus{name='COMMAND_LOGGER', key=1, minimum=0, maximum=1, desired=0, actual=0, requested=0, releasing=0, failed=0, started=0, startFailed=0, completed=0, failureMessage=''} : expected 0
2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG state.AppState (AppState.java:checkFailureThreshold(1620)) - Failure count of component: COMMAND_LOGGER: 0, threshold=5
2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,821 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 0
2014-11-12 09:29:35,821 [Shutdown] INFO  mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook executing
2014-11-12 09:29:35,822 [Shutdown] INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:50007
2014-11-12 09:29:35,825 [Shutdown] INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:52395
2014-11-12 09:29:35,829 [Shutdown] INFO  mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook complete
2014-11-12 09:29:35,834 [Thread-1] INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2014-11-12 09:29:36,037 [Thread-1] INFO  ipc.Server (Server.java:stop(2437)) - Stopping server on 40684
2014-11-12 09:29:36,039 [IPC Server listener on 40684] INFO  ipc.Server (Server.java:run(706)) - Stopping IPC Server listener on 40684
2014-11-12 09:29:36,039 [IPC Server Responder] INFO  ipc.Server (Server.java:run(832)) - Stopping IPC Server Responder
2014-11-12 09:29:36,040 [Thread-1] DEBUG actions.QueueService (QueueService.java:schedule(91)) - Scheduling org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
2014-11-12 09:29:36,040 [AMRM Callback Handler Thread] INFO  impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(276)) - Interrupted while waiting for queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2014-11-12 09:29:36,042 [AmExecutor-005] DEBUG actions.QueueService (QueueService.java:run(176)) - Propagating org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
2014-11-12 09:29:36,042 [AmExecutor-005] INFO  actions.QueueService (QueueService.java:run(179)) - QueueService processor terminated

Attachments

Activity

People

Assignee:: Steve Loughran

Reporter:: Gour Saha

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 13/Nov/14 01:06

Updated:: 25/Mar/15 20:28

Resolved:: 25/Mar/15 20:28