Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3561

Non-AM Containers continue to run even after AM is stopped

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: nodemanager, yarn
    • Labels:
      None
    • Environment:

      debian 7

      Description

      Non-AM containers continue to run even after application is stopped. This occurred while deploying Storm 0.9.3 using Slider (0.60.0 and 0.70.1) in a Hadoop 2.6 deployment.

      Following are the NM logs from 2 different nodes:
      host-07 - where Slider AM was running
      host-03 - where Storm NIMBUS container was running.

      Note: The logs are partial, starting with the time when the relevant Slider AM and NIMBUS containers were allocated, till the time when the Slider AM was stopped. Also, the large number of "Memory usage" log lines were removed keeping only a few starts and ends of every segment.

      NM log from host-07 where Slider AM container was running:

      2015-04-29 00:39:24,614 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(356)) - Stopping resource-monitoring for container_1428575950531_0020_02_000001
      2015-04-29 00:41:10,310 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:41:10,322 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(803)) - Start request for container_1428575950531_0021_01_000001 by user yarn
      2015-04-29 00:41:10,322 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(843)) - Creating a new application reference for app application_1428575950531_0021
      2015-04-29 00:41:10,323 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from NEW to INITING
      2015-04-29 00:41:10,325 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.105.162	OPERATION=Start Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000001
      2015-04-29 00:41:10,328 WARN  logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users.
      2015-04-29 00:41:10,328 WARN  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(182)) - rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished.
      2015-04-29 00:41:10,351 INFO  application.Application (ApplicationImpl.java:transition(304)) - Adding container_1428575950531_0021_01_000001 to application application_1428575950531_0021
      2015-04-29 00:41:10,352 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from INITING to RUNNING
      2015-04-29 00:41:10,356 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000001 transitioned from NEW to LOCALIZING
      2015-04-29 00:41:10,357 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1428575950531_0021
      2015-04-29 00:41:10,357 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/htrace-core-3.0.4.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,357 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jettison-1.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-util-1.0.0-M20.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/log4j-server.properties transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/httpcore-4.2.5.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-core-3.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,359 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slf4j-log4j12-1.6.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,359 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-env.sh transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,359 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-shuffle-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,359 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xercesImpl-2.9.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,360 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-hdfs-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,360 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-beanutils-1.7.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,360 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/stax-api-1.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,360 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/agent/slider-agent.tar.gz transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,361 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-registry-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,361 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xmlenc-0.52.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,361 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-core-asl-1.9.13.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,361 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-jvm-3.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,362 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/paranamer-2.3.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,362 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slider-core-0.70.1-incubating.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,362 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/log4j-1.2.17.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,362 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-client.xml transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,363 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-databind-2.2.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,363 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guice-servlet-3.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,363 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xml-apis-1.3.04.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,363 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/log4j.properties transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,364 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-server.xml transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,364 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jaxb-impl-2.2.3-1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,364 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-server-web-proxy-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,364 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-core-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,365 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-client-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,365 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-mapper-asl-1.9.13.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,365 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/asm-3.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,365 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-sslengine-6.1.26.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,366 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/zookeeper-3.4.6.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,366 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-common-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,366 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsp-api-2.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,366 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-auth-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,367 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-math3-3.1.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,367 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-core-1.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,367 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-api-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,367 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/avro-1.7.4.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,368 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-client-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,368 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsr311-api-1.1.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,368 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-core-2.2.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,368 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-framework-2.4.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,369 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/netty-3.7.0.Final.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,369 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-logging-1.1.3.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,369 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-lang-2.6.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,369 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-recipes-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,370 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-annotations-2.2.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,370 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsr305-1.3.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,370 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slf4j-api-1.7.5.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,370 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/aopalliance-1.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,371 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-digester-1.8.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,371 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-json-3.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,371 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-compress-1.4.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,371 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-jobclient-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,372 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-net-3.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,372 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-server-common-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,372 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-6.1.26.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,372 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jcommander-1.30.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,373 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/httpclient-4.2.5.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,373 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-common-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,373 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xz-1.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,373 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/apacheds-kerberos-codec-2.0.0-M15.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,374 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-beanutils-core-1.8.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,374 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-xc-1.9.13.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,374 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-healthchecks-3.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,374 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-io-2.4.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,375 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-common-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,375 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guava-11.0.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,375 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-collections-3.2.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,375 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-client-1.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,376 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/servlet-api-2.5.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,376 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/protobuf-java-2.5.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,376 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/snappy-java-1.0.4.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,376 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jline-0.9.94.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,377 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/apacheds-i18n-2.0.0-M15.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,377 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jaxb-api-2.2.7.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,377 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/leveldbjni-all-1.8.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,377 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/gson-2.2.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,378 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-app-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,378 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slider.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,378 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-jaxrs-1.9.13.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,378 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-annotations-2.6.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,379 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-json-1.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,379 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-httpclient-3.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,379 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-util-6.1.26.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,379 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-asn1-api-1.0.0-M20.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,380 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-server-1.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,380 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-cli-1.2.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,380 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-servlets-3.0.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,380 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-configuration-1.6.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,381 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-codec-1.6.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,381 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/javax.inject-1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,381 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-guice-1.9.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,381 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-client-2.4.1.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,382 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guice-3.0.jar transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:10,382 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(694)) - Created localizer for container_1428575950531_0021_01_000001
      2015-04-29 00:41:10,384 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1148)) - Writing credentials to the nmPrivate file /grid/2/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000001.tokens. Credentials list: 
      2015-04-29 00:41:10,494 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(606)) - Initializing user yarn
      2015-04-29 00:41:10,505 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(116)) - Copying from /grid/2/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000001.tokens to /grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001.tokens
      2015-04-29 00:41:10,505 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(123)) - Localizer CWD set to /grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021 = file:/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021
      2015-04-29 00:41:10,591 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/htrace-core-3.0.4.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/10/htrace-core-3.0.4.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,620 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jettison-1.1.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/11/jettison-1.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,652 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-util-1.0.0-M20.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/12/api-util-1.0.0-M20.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,679 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/log4j-server.properties(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/13/log4j-server.properties) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,707 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/httpcore-4.2.5.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/14/httpcore-4.2.5.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,733 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-core-3.0.1.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/15/metrics-core-3.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,760 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slf4j-log4j12-1.6.1.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/16/slf4j-log4j12-1.6.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,786 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-env.sh(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/17/slider-env.sh) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,813 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-shuffle-2.6.0.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/18/hadoop-mapreduce-client-shuffle-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,846 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xercesImpl-2.9.1.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/19/xercesImpl-2.9.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,920 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-hdfs-2.6.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/20/hadoop-hdfs-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,952 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-beanutils-1.7.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/21/commons-beanutils-1.7.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:10,979 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/stax-api-1.0.1.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/22/stax-api-1.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,135 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/agent/slider-agent.tar.gz(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/23/slider-agent.tar.gz) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,164 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-registry-2.6.0.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/24/hadoop-yarn-registry-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,192 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xmlenc-0.52.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/25/xmlenc-0.52.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,224 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-core-asl-1.9.13.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/26/jackson-core-asl-1.9.13.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,250 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-jvm-3.0.1.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/27/metrics-jvm-3.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,276 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/paranamer-2.3.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/28/paranamer-2.3.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,320 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slider-core-0.70.1-incubating.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/29/slider-core-0.70.1-incubating.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,355 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/log4j-1.2.17.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/30/log4j-1.2.17.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,381 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-client.xml(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/31/slider-client.xml) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,414 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-databind-2.2.2.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/32/jackson-databind-2.2.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,441 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guice-servlet-3.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/33/guice-servlet-3.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,468 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xml-apis-1.3.04.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/34/xml-apis-1.3.04.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,494 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/log4j.properties(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/35/log4j.properties) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,519 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/slider-server.xml(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/36/slider-server.xml) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,558 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jaxb-impl-2.2.3-1.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/37/jaxb-impl-2.2.3-1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,585 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-server-web-proxy-2.6.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/38/hadoop-yarn-server-web-proxy-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,620 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-core-2.6.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/39/hadoop-mapreduce-client-core-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,648 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-client-2.6.0.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/40/hadoop-yarn-client-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,685 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-mapper-asl-1.9.13.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/41/jackson-mapper-asl-1.9.13.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,712 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/asm-3.1.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/42/asm-3.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,740 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-sslengine-6.1.26.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/43/jetty-sslengine-6.1.26.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,779 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/zookeeper-3.4.6.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/44/zookeeper-3.4.6.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,811 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-common-2.6.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/45/hadoop-mapreduce-client-common-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,839 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsp-api-2.1.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/46/jsp-api-2.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,873 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-auth-2.6.0.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/47/hadoop-auth-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,909 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-math3-3.1.1.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/48/commons-math3-3.1.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,943 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-core-1.9.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/49/jersey-core-1.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:11,982 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-api-2.6.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/50/hadoop-yarn-api-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,015 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/avro-1.7.4.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/51/avro-1.7.4.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,042 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-client-2.6.0.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/52/hadoop-client-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,069 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsr311-api-1.1.1.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/53/jsr311-api-1.1.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,097 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-core-2.2.2.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/54/jackson-core-2.2.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,127 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-framework-2.4.1.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/55/curator-framework-2.4.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,172 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/netty-3.7.0.Final.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/56/netty-3.7.0.Final.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,218 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-logging-1.1.3.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/57/commons-logging-1.1.3.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,247 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-lang-2.6.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/58/commons-lang-2.6.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,276 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-recipes-2.6.0.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/59/curator-recipes-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,304 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-annotations-2.2.2.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/60/jackson-annotations-2.2.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,332 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jsr305-1.3.9.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/61/jsr305-1.3.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,359 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slf4j-api-1.7.5.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/62/slf4j-api-1.7.5.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,386 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/aopalliance-1.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/63/aopalliance-1.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,415 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-digester-1.8.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/64/commons-digester-1.8.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,442 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-json-3.0.1.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/65/metrics-json-3.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,473 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-compress-1.4.1.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/66/commons-compress-1.4.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,506 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-jobclient-2.6.0.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/67/hadoop-mapreduce-client-jobclient-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,536 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-net-3.1.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/68/commons-net-3.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,565 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-server-common-2.6.0.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/69/hadoop-yarn-server-common-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,601 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-6.1.26.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/70/jetty-6.1.26.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,629 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jcommander-1.30.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/71/jcommander-1.30.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,659 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/httpclient-4.2.5.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/72/httpclient-4.2.5.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,696 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-yarn-common-2.6.0.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/73/hadoop-yarn-common-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,727 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/xz-1.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/74/xz-1.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,765 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/apacheds-kerberos-codec-2.0.0-M15.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/75/apacheds-kerberos-codec-2.0.0-M15.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,794 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-beanutils-core-1.8.0.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/76/commons-beanutils-core-1.8.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,822 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-xc-1.9.13.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/77/jackson-xc-1.9.13.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,850 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-healthchecks-3.0.1.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/78/metrics-healthchecks-3.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,879 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-io-2.4.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/79/commons-io-2.4.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,927 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-common-2.6.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/80/hadoop-common-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:12,964 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guava-11.0.2.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/81/guava-11.0.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,001 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-collections-3.2.1.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/82/commons-collections-3.2.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,028 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-client-1.9.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/83/jersey-client-1.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,057 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/servlet-api-2.5.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/84/servlet-api-2.5.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,093 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/protobuf-java-2.5.0.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/85/protobuf-java-2.5.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,133 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/snappy-java-1.0.4.1.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/86/snappy-java-1.0.4.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,166 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jline-0.9.94.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/87/jline-0.9.94.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,194 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/apacheds-i18n-2.0.0-M15.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/88/apacheds-i18n-2.0.0-M15.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,223 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jaxb-api-2.2.7.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/89/jaxb-api-2.2.7.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,265 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/leveldbjni-all-1.8.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/90/leveldbjni-all-1.8.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,296 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/gson-2.2.2.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/91/gson-2.2.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,333 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-mapreduce-client-app-2.6.0.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/92/hadoop-mapreduce-client-app-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,369 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/slider.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/93/slider.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,397 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jackson-jaxrs-1.9.13.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/94/jackson-jaxrs-1.9.13.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,427 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/hadoop-annotations-2.6.0.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/95/hadoop-annotations-2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,457 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-json-1.9.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/96/jersey-json-1.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,487 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-httpclient-3.1.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/97/commons-httpclient-3.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,516 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jetty-util-6.1.26.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/98/jetty-util-6.1.26.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-asn1-api-1.0.0-M20.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/99/api-asn1-api-1.0.0-M20.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,576 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-server-1.9.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/100/jersey-server-1.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,606 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-cli-1.2.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/101/commons-cli-1.2.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,634 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/metrics-servlets-3.0.1.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/102/metrics-servlets-3.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,663 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-configuration-1.6.jar(->/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/103/commons-configuration-1.6.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,692 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/commons-codec-1.6.jar(->/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/104/commons-codec-1.6.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,721 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/javax.inject-1.jar(->/grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/105/javax.inject-1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,750 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jersey-guice-1.9.jar(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/106/jersey-guice-1.9.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,784 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/curator-client-2.4.1.jar(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/107/curator-client-2.4.1.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,816 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/guice-3.0.jar(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/108/guice-3.0.jar) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:13,816 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000001 transitioned from LOCALIZING to LOCALIZED
      2015-04-29 00:41:13,862 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000001 transitioned from LOCALIZED to RUNNING
      2015-04-29 00:41:13,870 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(267)) - launchContainer: [bash, /grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001/default_container_executor.sh]
      2015-04-29 00:41:15,618 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(346)) - Starting resource-monitoring for container_1428575950531_0021_01_000001
      2015-04-29 00:41:15,664 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 68.0 MB of 3.5 GB physical memory used; 739.0 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:18,681 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 172.5 MB of 3.5 GB physical memory used; 774.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:21,704 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 190.4 MB of 3.5 GB physical memory used; 799.8 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:23,617 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:41:23,627 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(803)) - Start request for container_1428575950531_0021_01_000005 by user yarn
      2015-04-29 00:41:23,629 INFO  application.Application (ApplicationImpl.java:transition(304)) - Adding container_1428575950531_0021_01_000005 to application application_1428575950531_0021
      2015-04-29 00:41:23,629 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.104.129	OPERATION=Start Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000005
      2015-04-29 00:41:23,629 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000005 transitioned from NEW to LOCALIZING
      2015-04-29 00:41:23,630 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1428575950531_0021
      2015-04-29 00:41:23,630 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/package/STORM/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:23,631 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(694)) - Created localizer for container_1428575950531_0021_01_000005
      2015-04-29 00:41:23,633 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1148)) - Writing credentials to the nmPrivate file /grid/4/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000005.tokens. Credentials list: 
      2015-04-29 00:41:23,755 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(606)) - Initializing user yarn
      2015-04-29 00:41:23,761 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(116)) - Copying from /grid/4/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000005.tokens to /grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000005.tokens
      2015-04-29 00:41:23,761 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(123)) - Localizer CWD set to /grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021 = file:/grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021
      2015-04-29 00:41:23,777 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:41:23,782 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatusInternal(1008)) - Getting container-status for container_1428575950531_0021_01_000005
      2015-04-29 00:41:23,782 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatusInternal(1022)) - Returning ContainerStatus: [ContainerId: container_1428575950531_0021_01_000005, State: RUNNING, Diagnostics: , ExitStatus: -1000, ]
      2015-04-29 00:41:24,251 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/package/STORM/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip(->/grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/109/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:24,251 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000005 transitioned from LOCALIZING to LOCALIZED
      2015-04-29 00:41:24,299 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000005 transitioned from LOCALIZED to RUNNING
      2015-04-29 00:41:24,307 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(267)) - launchContainer: [bash, /grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000005/default_container_executor.sh]
      2015-04-29 00:41:24,704 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(346)) - Starting resource-monitoring for container_1428575950531_0021_01_000005
      2015-04-29 00:41:24,747 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 210.6 MB of 3.5 GB physical memory used; 823.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:24,795 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 14.3 MB of 3.5 GB physical memory used; 279.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:27,837 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 211.4 MB of 3.5 GB physical memory used; 824.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:27,884 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 15.0 MB of 3.5 GB physical memory used; 280.0 MB of 7.3 GB virtual memory used
      .
      .
      .
      2015-04-29 00:55:55,558 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 222.8 MB of 3.5 GB physical memory used; 830.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:55:55,605 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:55:58,646 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 222.8 MB of 3.5 GB physical memory used; 830.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:55:58,693 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:00,463 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:56:00,467 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(953)) - Stopping container with container Id: container_1428575950531_0021_01_000005
      2015-04-29 00:56:00,468 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.104.129	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000005
      2015-04-29 00:56:00,468 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000005 transitioned from RUNNING to KILLING
      2015-04-29 00:56:00,469 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container container_1428575950531_0021_01_000005
      2015-04-29 00:56:01,741 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41169 for container-id container_1428575950531_0021_01_000001: 1.4 MB of 3.5 GB physical memory used; 10.9 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:01,763 INFO  launcher.ContainerLaunch (ContainerLaunch.java:call(346)) - Container container_1428575950531_0021_01_000001 succeeded 
      2015-04-29 00:56:01,763 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
      2015-04-29 00:56:01,764 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,782 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:01,816 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/1/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,816 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,818 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,819 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/4/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,819 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,820 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path : /grid/6/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,820 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	OPERATION=Container Finished - Succeeded	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000001
      2015-04-29 00:56:01,821 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
      2015-04-29 00:56:01,821 INFO  application.Application (ApplicationImpl.java:transition(347)) - Removing container_1428575950531_0021_01_000001 from application application_1428575950531_0021
      2015-04-29 00:56:01,822 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(488)) - Considering container container_1428575950531_0021_01_000001 for log-aggregation
      2015-04-29 00:56:01,822 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1428575950531_0021
      2015-04-29 00:56:02,490 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:56:02,496 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(953)) - Stopping container with container Id: container_1428575950531_0021_01_000001
      2015-04-29 00:56:02,497 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.105.162	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000001
      2015-04-29 00:56:02,499 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
      2015-04-29 00:56:04,782 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(356)) - Stopping resource-monitoring for container_1428575950531_0021_01_000001
      2015-04-29 00:56:04,824 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 41585 for container-id container_1428575950531_0021_01_000005: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      

      NM log from host-03 where NIMBUS container was running:

      2015-04-29 00:41:23,619 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:41:23,630 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(803)) - Start request for container_1428575950531_0021_01_000002 by user yarn
      2015-04-29 00:41:23,631 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(843)) - Creating a new application reference for app application_1428575950531_0021
      2015-04-29 00:41:23,631 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from NEW to INITING
      2015-04-29 00:41:23,632 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.104.129	OPERATION=Start Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000002
      2015-04-29 00:41:23,635 WARN  logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users.
      2015-04-29 00:41:23,636 WARN  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(182)) - rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished.
      2015-04-29 00:41:23,646 INFO  application.Application (ApplicationImpl.java:transition(304)) - Adding container_1428575950531_0021_01_000002 to application application_1428575950531_0021
      2015-04-29 00:41:23,647 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from INITING to RUNNING
      2015-04-29 00:41:23,647 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from NEW to LOCALIZING
      2015-04-29 00:41:23,648 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1428575950531_0021
      2015-04-29 00:41:23,648 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/agent/slider-agent.tar.gz transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:23,648 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/package/STORM/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip transitioned from INIT to DOWNLOADING
      2015-04-29 00:41:23,648 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(694)) - Created localizer for container_1428575950531_0021_01_000002
      2015-04-29 00:41:23,651 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1148)) - Writing credentials to the nmPrivate file /grid/2/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000002.tokens. Credentials list: 
      2015-04-29 00:41:23,766 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(606)) - Initializing user yarn
      2015-04-29 00:41:23,773 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(116)) - Copying from /grid/2/hadoop/yarn/local/nmPrivate/container_1428575950531_0021_01_000002.tokens to /grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000002.tokens
      2015-04-29 00:41:23,774 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(123)) - Localizer CWD set to /grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021 = file:/grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021
      2015-04-29 00:41:23,807 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:41:23,812 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatusInternal(1008)) - Getting container-status for container_1428575950531_0021_01_000002
      2015-04-29 00:41:23,812 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatusInternal(1022)) - Returning ContainerStatus: [ContainerId: container_1428575950531_0021_01_000002, State: RUNNING, Diagnostics: , ExitStatus: -1000, ]
      2015-04-29 00:41:23,991 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/agent/slider-agent.tar.gz(->/grid/2/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/10/slider-agent.tar.gz) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:24,344 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://zsexp/user/yarn/.slider/package/STORM/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip(->/grid/3/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/filecache/11/slider-storm-app-package-0.9.3.2.2.0.0-2041.zip) transitioned from DOWNLOADING to LOCALIZED
      2015-04-29 00:41:24,344 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from LOCALIZING to LOCALIZED
      2015-04-29 00:41:24,388 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from LOCALIZED to RUNNING
      2015-04-29 00:41:24,396 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(267)) - launchContainer: [bash, /grid/5/hadoop/yarn/local/usercache/yarn/appcache/application_1428575950531_0021/container_1428575950531_0021_01_000002/default_container_executor.sh]
      2015-04-29 00:41:26,947 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(346)) - Starting resource-monitoring for container_1428575950531_0021_01_000002
      2015-04-29 00:41:26,996 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.0 MB of 3.5 GB physical memory used; 280.0 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:30,038 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.0 MB of 3.5 GB physical memory used; 280.0 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:33,087 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.0 MB of 3.5 GB physical memory used; 280.0 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:36,129 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 28.2 MB of 3.5 GB physical memory used; 325.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:39,179 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 86.0 MB of 3.5 GB physical memory used; 63.3 GB of 7.3 GB virtual memory used
      2015-04-29 00:41:42,197 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 100.1 MB of 3.5 GB physical memory used; 31.8 GB of 7.3 GB virtual memory used
      2015-04-29 00:41:45,243 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 167.1 MB of 3.5 GB physical memory used; 31.8 GB of 7.3 GB virtual memory used
      2015-04-29 00:41:48,287 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 256.8 MB of 3.5 GB physical memory used; 1.8 GB of 7.3 GB virtual memory used
      2015-04-29 00:41:51,336 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.3 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:41:54,379 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.3 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      .
      .
      .
      2015-04-29 00:55:51,675 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:55:54,718 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:55:57,760 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:00,482 INFO  ipc.Server (Server.java:saslProcess(1306)) - Auth successful for appattempt_1428575950531_0021_000001 (auth:SIMPLE)
      2015-04-29 00:56:00,486 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(953)) - Stopping container with container Id: container_1428575950531_0021_01_000002
      2015-04-29 00:56:00,487 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.104.129	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000002
      2015-04-29 00:56:00,487 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from RUNNING to KILLING
      2015-04-29 00:56:00,489 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container container_1428575950531_0021_01_000002
      2015-04-29 00:56:00,802 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:02,494 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
      2015-04-29 00:56:03,849 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:06,892 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:09,940 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:12,983 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:16,032 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:19,075 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:22,122 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:25,164 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:28,211 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:31,253 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:34,302 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:37,344 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:40,392 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:43,434 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:46,482 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:49,523 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:52,570 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:55,613 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:56:58,661 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:01,704 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:04,752 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:07,794 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:10,843 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:13,886 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 280.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:16,935 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 296.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:19,977 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 296.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:23,025 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.6 MB of 3.5 GB physical memory used; 296.5 MB of 7.3 GB virtual memory used
      2015-04-29 00:57:26,067 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.7 MB of 3.5 GB physical memory used; 296.5 MB of 7.3 GB virtual memory used
      

      YARN properties of the cluster:

      {
          "description": "Complete slider application settings", 
          "updated": 0, 
          "entries": {
              "dfs.datanode.data.dir": "/grid/1/dfs/data,/grid/2/dfs/data,/grid/3/dfs/data,/grid/4/dfs/data,/grid/5/dfs/data,/grid/6/dfs/data", 
              "dfs.namenode.checkpoint.txns": "1000000", 
              "s3.replication": "3", 
              "mapreduce.output.fileoutputformat.compress.type": "BLOCK", 
              "mapreduce.jobtracker.jobhistory.lru.cache.size": "5", 
              "dfs.datanode.failed.volumes.tolerated": "0", 
              "hadoop.http.filter.initializers": "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer", 
              "mapreduce.cluster.temp.dir": "/tmp/hadoop-yarn-${hue.suffix}/mapred/temp", 
              "mapreduce.reduce.shuffle.memory.limit.percent": "0.25", 
              "yarn.nodemanager.keytab": "/etc/krb5.keytab", 
              "dfs.namenode.checkpoint.max-retries": "3", 
              "nfs.mountd.port": "4242", 
              "hadoop.registry.zk.retry.times": "5", 
              "yarn.resourcemanager.zk-acl": "world:anyone:rwcda", 
              "mapreduce.reduce.skip.maxgroups": "0", 
              "dfs.https.server.keystore.resource": "ssl-server.xml", 
              "yarn.app.mapreduce.task.container.log.backups": "0", 
              "dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket", 
              "hadoop.http.authentication.kerberos.keytab": "/home/yarn/hadoop.keytab", 
              "yarn.timeline-service.generic-application-history.store-class": "org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore", 
              "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage": "90", 
              "mapreduce.cluster.administrators": " hadoop", 
              "dfs.client.block.write.replace-datanode-on-failure.best-effort": "false", 
              "mapreduce.jobhistory.done-dir": "/mr-history/done", 
              "yarn.nodemanager.localizer.client.thread-count": "5", 
              "ha.failover-controller.new-active.rpc-timeout.ms": "60000", 
              "mapreduce.framework.name": "yarn", 
              "ha.health-monitor.check-interval.ms": "1000", 
              "io.file.buffer.size": "131072", 
              "mapreduce.shuffle.max.connections": "0", 
              "dfs.namenode.resource.check.interval": "5000", 
              "dfs.namenode.path.based.cache.block.map.allocation.percent": "0.25", 
              "dfs.namenode.checkpoint.period": "21600", 
              "mapreduce.task.tmp.dir": "./tmp", 
              "ipc.client.kill.max": "10", 
              "yarn.nodemanager.log-aggregation.debug-enabled": "false", 
              "dfs.client.mmap.cache.timeout.ms": "3600000", 
              "yarn.resourcemanager.scheduler.class": "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler", 
              "mapreduce.jobtracker.taskcache.levels": "2", 
              "s3.stream-buffer-size": "4096", 
              "dfs.namenode.secondary.http-address": "host2:50090", 
              "yarn.client.nodemanager-connect.max-wait-ms": "900000", 
              "dfs.namenode.decommission.interval": "30", 
              "dfs.namenode.http-address": "host1:50070", 
              "dfs.encrypt.data.transfer": "false", 
              "mapreduce.task.files.preserve.failedtasks": "false", 
              "yarn.resourcemanager.bind-host": "0.0.0.0", 
              "yarn.resourcemanager.ha.enabled": "false", 
              "dfs.datanode.address": "0.0.0.0:50010", 
              "dfs.namenode.avoid.write.stale.datanode": "true", 
              "hadoop.http.authentication.token.validity": "36000", 
              "yarn.resourcemanager.nodes.exclude-path": "/etc/hadoop/conf/yarn.exclude", 
              "hadoop.security.group.mapping.ldap.search.filter.group": "(objectClass=group)", 
              "dfs.client.failover.max.attempts": "15", 
              "yarn.resourcemanager.scheduler.monitor.policies": "org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy", 
              "hadoop.registry.zk.connection.timeout.ms": "15000", 
              "hadoop.security.crypto.cipher.suite": "AES/CTR/NoPadding", 
              "yarn.timeline-service.http-authentication.simple.anonymous.allowed": "true", 
              "mapreduce.task.profile.params": "-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s", 
              "yarn.resourcemanager.fs.state-store.retry-policy-spec": "2000, 500", 
              "yarn.admin.acl": "*", 
              "yarn.nodemanager.local-cache.max-files-per-directory": "8192", 
              "yarn.client.failover-retries-on-socket-timeouts": "0", 
              "dfs.namenode.retrycache.expirytime.millis": "600000", 
              "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms": "1000", 
              "dfs.client.failover.connection.retries.on.timeouts": "0", 
              "yarn.client.failover-proxy-provider": "org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider", 
              "mapreduce.map.sort.spill.percent": "0.7", 
              "file.stream-buffer-size": "4096", 
              "dfs.webhdfs.enabled": "true", 
              "ipc.client.connection.maxidletime": "0", 
              "mapreduce.task.combine.progress.records": "10000", 
              "mapreduce.jobtracker.persist.jobstatus.hours": "1", 
              "dfs.image.transfer.chunksize": "65536", 
              "yarn.nodemanager.address": "0.0.0.0:45454", 
              "dfs.datanode.ipc.address": "0.0.0.0:8010", 
              "yarn.resourcemanager.ha.automatic-failover.embedded": "true", 
              "mapreduce.jobhistory.recovery.store.fs.uri": "/tmp/hadoop-yarn-${hue.suffix}/mapred/history/recoverystore", 
              "dfs.namenode.http-address.$cluster_name.nn1": "host1:50070", 
              "yarn.resourcemanager.zk-state-store.parent-path": "/rmstore", 
              "dfs.namenode.http-address.$cluster_name.nn2": "host2:50070", 
              "yarn.app.mapreduce.am.job.task.listener.thread-count": "30", 
              "nfs.dump.dir": "/tmp/.hdfs-nfs", 
              "dfs.namenode.list.cache.pools.num.responses": "100", 
              "dfs.client.read.shortcircuit": "true", 
              "dfs.namenode.shared.edits.dir": "qjournal://host2:8485;host3:8485;host1:8485/$cluster_name", 
              "dfs.namenode.safemode.extension": "30000", 
              "ha.zookeeper.parent-znode": "/hadoop-ha", 
              "yarn.nodemanager.container-executor.class": "org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor", 
              "slider.config.loaded": "true", 
              "io.skip.checksum.errors": "false", 
              "dfs.namenode.path.based.cache.refresh.interval.ms": "30000", 
              "dfs.encrypt.data.transfer.cipher.key.bitlength": "128", 
              "yarn.resourcemanager.scheduler.client.thread-count": "50", 
              "yarn.nodemanager.recovery.dir": "/var/log/hadoop-yarn/nodemanager/recovery-state", 
              "mapreduce.job.emit-timeline-data": "false", 
              "hadoop.http.authentication.kerberos.principal": "HTTP/_HOST@LOCALHOST", 
              "mapreduce.reduce.log.level": "INFO", 
              "yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern": "^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$", 
              "fs.s3.maxRetries": "4", 
              "yarn.nodemanager.log-aggregation.num-log-files-per-app": "30", 
              "mapreduce.admin.reduce.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041", 
              "yarn.nodemanager.resourcemanager.minimum.version": "NONE", 
              "yarn.log-aggregation.retain-check-interval-seconds": "-1", 
              "hadoop.kerberos.kinit.command": "kinit", 
              "yarn.node-labels.fs-store.root-dir": "/system/yarn/node-labels", 
              "yarn.nodemanager.linux-container-executor.cgroups.mount": "false", 
              "yarn.nodemanager.process-kill-wait.ms": "2000", 
              "mapreduce.jobtracker.handler.count": "10", 
              "dfs.namenode.name.dir.restore": "true", 
              "mapreduce.jobhistory.admin.address": "0.0.0.0:10033", 
              "yarn.app.mapreduce.client-am.ipc.max-retries": "3", 
              "yarn.app.mapreduce.am.admin-command-opts": "-Dhdp.version=2.2.0.0-2041", 
              "hadoop.proxyuser.hue.hosts": "*", 
              "dfs.client.use.datanode.hostname": "false", 
              "dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction": "0.75f", 
              "hadoop.util.hash.type": "murmur", 
              "io.seqfile.lazydecompress": "true", 
              "dfs.datanode.dns.interface": "default", 
              "dfs.namenode.lazypersist.file.scrub.interval.sec": "300", 
              "yarn.client.max-cached-nodemanagers-proxies": "0", 
              "yarn.nodemanager.disk-health-checker.min-healthy-disks": "0.25", 
              "mapreduce.job.maxtaskfailures.per.tracker": "3", 
              "mapreduce.tasktracker.healthchecker.script.timeout": "600000", 
              "hadoop.security.group.mapping.ldap.search.attr.group.name": "cn", 
              "hadoop.security.crypto.buffer.size": "8192", 
              "fs.df.interval": "60000", 
              "dfs.client.cached.conn.retry": "3", 
              "dfs.namenode.kerberos.internal.spnego.principal": "${dfs.web.authentication.kerberos.principal}", 
              "mapreduce.job.reduce.shuffle.consumer.plugin.class": "org.apache.hadoop.mapreduce.task.reduce.Shuffle", 
              "mapreduce.jobtracker.address": "local", 
              "mapreduce.tasktracker.tasks.sleeptimebeforesigkill": "5000", 
              "dfs.journalnode.rpc-address": "0.0.0.0:8485", 
              "dfs.namenode.fs-limits.max-blocks-per-file": "1048576", 
              "mapreduce.job.acl-view-job": " ", 
              "dfs.client.block.write.replace-datanode-on-failure.policy": "DEFAULT", 
              "yarn.app.mapreduce.am.job.committer.cancel-timeout": "60000", 
              "mapreduce.shuffle.connection-keep-alive.enable": "false", 
              "dfs.namenode.replication.interval": "3", 
              "dfs.namenode.num.checkpoints.retained": "2", 
              "mapreduce.jobhistory.minicluster.fixed.ports": "false", 
              "mapreduce.jobhistory.admin.acl": "*", 
              "mapreduce.tasktracker.http.address": "0.0.0.0:50060", 
              "yarn.resourcemanager.scheduler.address": "host2:8030", 
              "dfs.datanode.directoryscan.threads": "1", 
              "hadoop.security.group.mapping.ldap.ssl": "false", 
              "mapreduce.reduce.memory.mb": "3584", 
              "dfs.http.policy": "HTTP_ONLY", 
              "mapreduce.task.merge.progress.records": "10000", 
              "yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size": "10", 
              "dfs.heartbeat.interval": "3", 
              "yarn.resourcemanager.recovery.enabled": "false", 
              "dfs.hosts.exclude": "/etc/hadoop/conf/dfs.exclude", 
              "net.topology.script.number.args": "100", 
              "mapreduce.local.clientfactory.class.name": "org.apache.hadoop.mapred.LocalClientFactory", 
              "dfs.client-write-packet-size": "65536", 
              "hadoop.security.group.mapping.ldap.directory.search.timeout": "10000", 
              "dfs.nameservices": "$cluster_name", 
              "io.native.lib.available": "true", 
              "dfs.client.failover.connection.retries": "0", 
              "yarn.nodemanager.disk-health-checker.interval-ms": "120000", 
              "dfs.blocksize": "134217728", 
              "dfs.client.use.legacy.blockreader.local": "false", 
              "yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs": "86400", 
              "fs.s3a.connection.ssl.enabled": "true", 
              "mapreduce.jobhistory.webapp.address": "host2:19888", 
              "yarn.client.failover-retries": "0", 
              "yarn.resourcemanager.resource-tracker.client.thread-count": "50", 
              "hadoop.registry.jaas.context": "Client", 
              "dfs.blockreport.initialDelay": "120", 
              "yarn.resourcemanager.zk-timeout-ms": "10000", 
              "ha.health-monitor.rpc-timeout.ms": "45000", 
              "yarn.nodemanager.aux-services.mapreduce_shuffle.class": "org.apache.hadoop.mapred.ShuffleHandler", 
              "mapreduce.reduce.markreset.buffer.percent": "0.0", 
              "dfs.ha.tail-edits.period": "60", 
              "yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size": "10000", 
              "mapreduce.admin.user.env": "LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-2041/hadoop/lib/native:/usr/hdp/2.2.0.0-2041/hadoop/lib/native/Linux-amd64-64", 
              "yarn.nodemanager.health-checker.script.timeout-ms": "60000", 
              "yarn.resourcemanager.client.thread-count": "50", 
              "file.bytes-per-checksum": "512", 
              "dfs.replication.max": "50", 
              "dfs.namenode.max.extra.edits.segments.retained": "10000", 
              "yarn.nodemanager.linux-container-executor.group": "hadoop", 
              "io.map.index.skip": "0", 
              "mapreduce.jobhistory.bind-host": "0.0.0.0", 
              "yarn.timeline-service.client.retry-interval-ms": "1000", 
              "mapreduce.task.timeout": "300000", 
              "mapreduce.reduce.cpu.vcores": "1", 
              "dfs.datanode.du.reserved": "1073741824", 
              "dfs.support.append": "true", 
              "ftp.blocksize": "67108864", 
              "dfs.client.file-block-storage-locations.num-threads": "10", 
              "yarn.nodemanager.container-manager.thread-count": "20", 
              "ipc.server.listen.queue.size": "128", 
              "hadoop.ssl.hostname.verifier": "DEFAULT", 
              "yarn.resourcemanager.ha.automatic-failover.enabled": "true", 
              "nfs.server.port": "2049", 
              "mapreduce.tasktracker.dns.interface": "default", 
              "hadoop.security.group.mapping.ldap.search.attr.member": "member", 
              "mapreduce.job.userlog.retain.hours": "24", 
              "mapreduce.tasktracker.outofband.heartbeat": "false", 
              "fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem", 
              "yarn.log.server.url": "http://host2:19888/jobhistory/logs", 
              "hadoop.registry.system.acls": "sasl:yarn@, sasl:mapred@, sasl:mapred@hdfs@", 
              "yarn.nodemanager.resource.memory-mb": "96768", 
              "yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds": "-1", 
              "dfs.webhdfs.user.provider.user.pattern": "^[A-Za-z_][A-Za-z0-9._-]*[$]?$", 
              "dfs.namenode.delegation.token.renew-interval": "86400000", 
              "hadoop.ssl.keystores.factory.class": "org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory", 
              "hadoop.registry.zk.retry.ceiling.ms": "60000", 
              "yarn.http.policy": "HTTP_ONLY", 
              "dfs.datanode.sync.behind.writes": "false", 
              "nfs.wtmax": "1048576", 
              "fs.AbstractFileSystem.har.impl": "org.apache.hadoop.fs.HarFs", 
              "dfs.client.read.shortcircuit.skip.checksum": "false", 
              "hadoop.security.random.device.file.path": "/dev/urandom", 
              "mapreduce.map.maxattempts": "4", 
              "yarn.timeline-service.webapp.address": "host2:8188", 
              "dfs.datanode.handler.count": "10", 
              "hadoop.ssl.require.client.cert": "false", 
              "ftp.client-write-packet-size": "65536", 
              "dfs.client.write.exclude.nodes.cache.expiry.interval.millis": "600000", 
              "ipc.server.tcpnodelay": "true", 
              "mapreduce.jobhistory.cleaner.enable": "true", 
              "fs.du.interval": "600000", 
              "mapreduce.reduce.shuffle.retry-delay.max.ms": "60000", 
              "mapreduce.task.profile.reduces": "0-2", 
              "ha.health-monitor.connect-retry-interval.ms": "1000", 
              "hadoop.fuse.connection.timeout": "300", 
              "dfs.permissions.superusergroup": "hdfs", 
              "mapreduce.jobtracker.jobhistory.task.numberprogresssplits": "12", 
              "fs.ftp.host.port": "21", 
              "mapreduce.map.speculative": "false", 
              "dfs.datanode.data.dir.perm": "750", 
              "mapreduce.client.submit.file.replication": "10", 
              "dfs.namenode.startup.delay.block.deletion.sec": "3600", 
              "s3native.blocksize": "67108864", 
              "mapreduce.job.ubertask.maxmaps": "9", 
              "dfs.namenode.replication.min": "1", 
              "mapreduce.cluster.acls.enabled": "false", 
              "hadoop.security.uid.cache.secs": "14400", 
              "nfs.allow.insecure.ports": "true", 
              "yarn.nodemanager.localizer.fetch.thread-count": "4", 
              "map.sort.class": "org.apache.hadoop.util.QuickSort", 
              "hadoop.proxyuser.hue.groups": "*", 
              "fs.trash.checkpoint.interval": "0", 
              "hadoop.proxyuser.hcat.groups": "users", 
              "dfs.image.transfer.timeout": "60000", 
              "dfs.namenode.name.dir": "/grid/1/dfs/nn,/grid/2/dfs/nn,/grid/3/dfs/nn,/grid/4/dfs/nn,/grid/5/dfs/nn,/grid/6/dfs/nn", 
              "ipc.client.connect.timeout": "20000", 
              "yarn.app.mapreduce.am.staging-dir": "/user", 
              "fs.AbstractFileSystem.file.impl": "org.apache.hadoop.fs.local.LocalFs", 
              "yarn.nodemanager.env-whitelist": "JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME", 
              "hadoop.registry.zk.retry.interval.ms": "1000", 
              "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage": "false", 
              "yarn.timeline-service.keytab": "/etc/krb5.keytab", 
              "dfs.image.compression.codec": "org.apache.hadoop.io.compress.DefaultCodec", 
              "mapreduce.job.reduces": "1", 
              "mapreduce.job.complete.cancel.delegation.tokens": "true", 
              "mapreduce.jobhistory.recovery.store.class": "org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService", 
              "hadoop.security.group.mapping.ldap.search.filter.user": "(&(objectClass=user)(sAMAccountName={0}))", 
              "dfs.namenode.enable.retrycache": "true", 
              "yarn.nodemanager.sleep-delay-before-sigkill.ms": "250", 
              "mapreduce.jobhistory.joblist.cache.size": "20000", 
              "mapreduce.tasktracker.healthchecker.interval": "60000", 
              "mapreduce.jobtracker.heartbeats.in.second": "100", 
              "hadoop.security.auth_to_local": " RULE:[2:$1@$0]([rn]m@.*)s/.*/yarn/ RULE:[2:$1@$0](jhs@.*)s/.*/mapred/ RULE:[2:$1@$0]([nd]n@.*)s/.*/hdfs/ RULE:[2:$1@$0](hm@.*)s/.*/hbase/ RULE:[2:$1@$0](rs@.*)s/.*/hbase/ DEFAULT", 
              "mapreduce.admin.map.child.java.opts": "-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041", 
              "mapreduce.jobtracker.persist.jobstatus.dir": "/jobtracker/jobsInfo", 
              "dfs.namenode.backup.http-address": "0.0.0.0:50105", 
              "hadoop.rpc.protection": "authentication", 
              "dfs.client.mmap.enabled": "true", 
              "yarn.app.mapreduce.am.container.log.backups": "0", 
              "ftp.stream-buffer-size": "4096", 
              "dfs.namenode.https-address": "host1:50470", 
              "yarn.timeline-service.address": "host2:10200", 
              "dfs.ha.log-roll.period": "120", 
              "yarn.nodemanager.recovery.enabled": "true", 
              "hadoop.security.groups.negative-cache.secs": "30", 
              "yarn.resourcemanager.admin.client.thread-count": "1", 
              "dfs.datanode.fsdatasetcache.max.threads.per.volume": "4", 
              "file.client-write-packet-size": "65536", 
              "hadoop.http.authentication.simple.anonymous.allowed": "true", 
              "yarn.timeline-service.leveldb-timeline-store.path": "/grid/1/hadoop/yarn/timeline", 
              "dfs.namenode.https-address.$cluster_name.nn1": "host1:50470", 
              "yarn.resourcemanager.proxy-user-privileges.enabled": "false", 
              "dfs.datanode.drop.cache.behind.reads": "false", 
              "yarn.nodemanager.log.retain-seconds": "10800", 
              "dfs.image.transfer.bandwidthPerSec": "0", 
              "yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms": "10000", 
              "hadoop.registry.zk.session.timeout.ms": "60000", 
              "dfs.datanode.slow.io.warning.threshold.ms": "300", 
              "mapreduce.tasktracker.instrumentation": "org.apache.hadoop.mapred.TaskTrackerMetricsInst", 
              "ha.failover-controller.cli-check.rpc-timeout.ms": "20000", 
              "dfs.namenode.https-address.$cluster_name.nn2": "host2:50470", 
              "yarn.nodemanager.linux-container-executor.cgroups.hierarchy": "hadoop-yarn", 
              "dfs.namenode.write.stale.datanode.ratio": "1.0f", 
              "hadoop.security.groups.cache.warn.after.ms": "5000", 
              "mapreduce.reduce.shuffle.fetch.retry.timeout-ms": "30000", 
              "mapreduce.jobhistory.client.thread-count": "10", 
              "io.mapfile.bloom.size": "1048576", 
              "yarn.resourcemanager.work-preserving-recovery.enabled": "true", 
              "dfs.ha.fencing.ssh.connect-timeout": "30000", 
              "yarn.resourcemanager.zk-num-retries": "1000", 
              "hadoop.registry.zk.root": "/registry", 
              "s3.bytes-per-checksum": "512", 
              "yarn.app.mapreduce.am.container.log.limit.kb": "0", 
              "dfs.namenode.edit.log.autoroll.check.interval.ms": "300000", 
              "fs.automatic.close": "true", 
              "yarn.node-labels.fs-store.retry-policy-spec": "2000, 500", 
              "fs.trash.interval": "360", 
              "dfs.journalnode.https-address": "0.0.0.0:8481", 
              "yarn.timeline-service.ttl-ms": "2678400000", 
              "hadoop.security.authentication": "simple", 
              "fs.defaultFS": "hdfs://$cluster_name", 
              "nfs.rtmax": "1048576", 
              "hadoop.ssl.server.conf": "ssl-server.xml", 
              "ipc.client.connect.max.retries": "50", 
              "yarn.resourcemanager.delayed.delegation-token.removal-interval-ms": "30000", 
              "dfs.journalnode.http-address": "0.0.0.0:8480", 
              "dfs.namenode.xattrs.enabled": "true", 
              "dfs.datanode.shared.file.descriptor.paths": "/dev/shm,/tmp", 
              "mapreduce.jobtracker.taskscheduler": "org.apache.hadoop.mapred.JobQueueTaskScheduler", 
              "mapreduce.job.speculative.speculativecap": "0.1", 
              "yarn.timeline-service.store-class": "org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore", 
              "yarn.am.liveness-monitor.expiry-interval-ms": "600000", 
              "mapreduce.output.fileoutputformat.compress": "false", 
              "yarn.timeline-service.bind-host": "0.0.0.0", 
              "dfs.user.home.dir.prefix": "/user", 
              "yarn.app.mapreduce.am.log.level": "INFO", 
              "net.topology.node.switch.mapping.impl": "org.apache.hadoop.net.ScriptBasedMapping", 
              "dfs.namenode.replication.considerLoad": "true", 
              "dfs.namenode.fs-limits.min-block-size": "1048576", 
              "fs.swift.impl": "org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem", 
              "dfs.namenode.audit.loggers": "default", 
              "yarn.nodemanager.bind-host": "0.0.0.0", 
              "mapreduce.job.max.split.locations": "10", 
              "yarn.resourcemanager.address": "host2:8050", 
              "mapreduce.job.counters.max": "120", 
              "mapreduce.reduce.shuffle.fetch.retry.enabled": "1", 
              "dfs.client.block.write.retries": "3", 
              "dfs.short.circuit.shared.memory.watcher.interrupt.check.ms": "60000", 
              "dfs.namenode.resource.checked.volumes.minimum": "1", 
              "io.map.index.interval": "128", 
              "mapred.child.java.opts": "-Xmx200m", 
              "mapreduce.tasktracker.local.dir.minspacestart": "0", 
              "mapreduce.client.progressmonitor.pollinterval": "1000", 
              "dfs.client.https.keystore.resource": "ssl-client.xml", 
              "mapreduce.task.profile.map.params": "-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s", 
              "mapreduce.jobtracker.tasktracker.maxblacklists": "4", 
              "mapreduce.job.queuename": "default", 
              "hadoop.registry.zk.quorum": "host2:2181", 
              "yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts": "3", 
              "yarn.nodemanager.localizer.address": "0.0.0.0:8040", 
              "io.mapfile.bloom.error.rate": "0.005", 
              "yarn.nodemanager.delete.thread-count": "4", 
              "mapreduce.job.split.metainfo.maxsize": "10000000", 
              "yarn.scheduler.maximum-allocation-vcores": "32", 
              "dfs.https.port": "50470", 
              "yarn.app.mapreduce.am.resource.mb": "3584", 
              "dfs.datanode.dns.nameserver": "default", 
              "dfs.client.slow.io.warning.threshold.ms": "30000", 
              "mapreduce.job.reducer.preempt.delay.sec": "0", 
              "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb": "1000", 
              "mapreduce.map.output.compress.codec": "org.apache.hadoop.io.compress.DefaultCodec", 
              "dfs.namenode.accesstime.precision": "0", 
              "mapreduce.map.log.level": "INFO", 
              "fs.s3a.connection.maximum": "15", 
              "io.seqfile.compress.blocksize": "1000000", 
              "mapreduce.tasktracker.taskcontroller": "org.apache.hadoop.mapred.DefaultTaskController", 
              "hadoop.security.groups.cache.secs": "300", 
              "dfs.datanode.cache.revocation.timeout.ms": "900000", 
              "dfs.client.context": "default", 
              "hadoop.proxyuser.hive.groups": "users", 
              "mapreduce.input.lineinputformat.linespermap": "1", 
              "mapreduce.job.end-notification.max.attempts": "5", 
              "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user": "nobody", 
              "yarn.nodemanager.webapp.address": "0.0.0.0:8042", 
              "mapreduce.jobhistory.recovery.enable": "false", 
              "mapreduce.jobtracker.expire.trackers.interval": "600000", 
              "yarn.resourcemanager.webapp.address": "host2:8088", 
              "hadoop.security.kms.client.encrypted.key.cache.num.refill.threads": "2", 
              "yarn.nodemanager.health-checker.interval-ms": "135000", 
              "mapreduce.jobhistory.loadedjobs.cache.size": "5", 
              "hadoop.security.authorization": "false", 
              "mapreduce.job.map.output.collector.class": "org.apache.hadoop.mapred.MapTask$MapOutputBuffer", 
              "mapreduce.am.max-attempts": "2", 
              "fs.ftp.host": "0.0.0.0", 
              "fs.s3a.attempts.maximum": "10", 
              "yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms": "1000", 
              "mapreduce.ifile.readahead": "true", 
              "yarn.resourcemanager.scheduler.monitor.enable": "false", 
              "yarn.resourcemanager.zk-retry-interval-ms": "1000", 
              "ha.zookeeper.session-timeout.ms": "5000", 
              "rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB": "org.apache.hadoop.ipc.ProtobufRpcEngine", 
              "mapreduce.tasktracker.taskmemorymanager.monitoringinterval": "5000", 
              "mapreduce.reduce.shuffle.parallelcopies": "30", 
              "mapreduce.map.skip.maxrecords": "0", 
              "dfs.client.mmap.retry.timeout.ms": "300000", 
              "dfs.namenode.avoid.read.stale.datanode": "true", 
              "yarn.timeline-service.webapp.https.address": "host2:8190", 
              "dfs.https.enable": "false", 
              "mapreduce.reduce.shuffle.read.timeout": "180000", 
              "dfs.namenode.list.encryption.zones.num.responses": "100", 
              "mapreduce.jobtracker.instrumentation": "org.apache.hadoop.mapred.JobTrackerMetricsInst", 
              "mapreduce.output.fileoutputformat.compress.codec": "org.apache.hadoop.io.compress.DefaultCodec", 
              "yarn.nodemanager.remote-app-log-dir-suffix": "logs", 
              "ipc.client.connect.retry.interval": "1000", 
              "dfs.blockreport.intervalMsec": "21600000", 
              "mapreduce.reduce.speculative": "false", 
              "mapreduce.jobhistory.keytab": "/etc/security/keytab/jhs.service.keytab", 
              "mapreduce.jobhistory.datestring.cache.size": "200000", 
              "dfs.datanode.balance.bandwidthPerSec": "6250000", 
              "file.blocksize": "67108864", 
              "yarn.resourcemanager.admin.address": "host2:8141", 
              "mapreduce.map.cpu.vcores": "1", 
              "yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled": "false", 
              "yarn.resourcemanager.resource-tracker.address": "host2:8025", 
              "yarn.resourcemanager.configuration.provider-class": "org.apache.hadoop.yarn.LocalConfigurationProvider", 
              "mapreduce.tasktracker.local.dir.minspacekill": "0", 
              "mapreduce.jobtracker.staging.root.dir": "/tmp/hadoop-yarn-${hue.suffix}/mapred/staging", 
              "mapreduce.jobtracker.retiredjobs.cache.size": "1000", 
              "hadoop.registry.rm.enabled": "true", 
              "dfs.ha.fencing.methods": "shell(/bin/true)", 
              "ipc.client.connect.max.retries.on.timeouts": "45", 
              "hadoop.security.crypto.codec.classes.aes.ctr.nopadding": "org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec", 
              "ha.zookeeper.acl": "world:anyone:rwcda", 
              "mapreduce.app-submission.cross-platform": "false", 
              "yarn.nodemanager.local-dirs": "/grid/1/hadoop/yarn/local,/grid/2/hadoop/yarn/local,/grid/3/hadoop/yarn/local,/grid/4/hadoop/yarn/local,/grid/5/hadoop/yarn/local,/grid/6/hadoop/yarn/local", 
              "rpc.metrics.quantile.enable": "false", 
              "dfs.block.access.key.update.interval": "600", 
              "mapreduce.reduce.shuffle.connect.timeout": "180000", 
              "dfs.block.access.token.lifetime": "600", 
              "mapreduce.job.end-notification.retry.attempts": "0", 
              "dfs.namenode.fs-limits.max-xattrs-per-inode": "32", 
              "mapreduce.jobtracker.system.dir": "/tmp/hadoop-yarn-${hue.suffix}/mapred/system", 
              "dfs.client.file-block-storage-locations.timeout.millis": "1000", 
              "yarn.nodemanager.admin-env": "MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX", 
              "yarn.log-aggregation.retain-seconds": "2592000", 
              "mapreduce.jobtracker.jobhistory.block.size": "3145728", 
              "yarn.timeline-service.handler-thread-count": "10", 
              "mapreduce.tasktracker.indexcache.mb": "10", 
              "dfs.namenode.checkpoint.check.period": "60", 
              "dfs.client.block.write.replace-datanode-on-failure.enable": "true", 
              "yarn.resourcemanager.hostname": "host2", 
              "dfs.datanode.directoryscan.interval": "21600", 
              "net.topology.impl": "org.apache.hadoop.net.NetworkTopology", 
              "fs.s3a.multipart.purge.age": "86400", 
              "dfs.client.failover.proxy.provider.$cluster_name": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", 
              "hadoop.security.java.secure.random.algorithm": "SHA1PRNG", 
              "yarn.nodemanager.container-monitor.interval-ms": "3000", 
              "dfs.default.chunk.view.size": "32768", 
              "fs.s3a.multipart.threshold": "2147483647", 
              "mapreduce.job.speculative.slownodethreshold": "1.0", 
              "mapreduce.job.reduce.slowstart.completedmaps": "0.05", 
              "hadoop.security.instrumentation.requires.admin": "false", 
              "io.compression.codec.bzip2.library": "system-native", 
              "dfs.namenode.safemode.min.datanodes": "0", 
              "hadoop.registry.secure": "false", 
              "hadoop.http.authentication.signature.secret.file": "/home/yarn/hadoop-http-auth-signature-secret", 
              "mapreduce.reduce.maxattempts": "4", 
              "yarn.nodemanager.localizer.cache.target-size-mb": "10240", 
              "s3native.replication": "3", 
              "dfs.datanode.https.address": "0.0.0.0:50475", 
              "dfs.journalnode.edits.dir": "/grid/1/dfs/journal,/grid/2/dfs/journal,/grid/3/dfs/journal,/grid/4/dfs/journal,/grid/5/dfs/journal,/grid/6/dfs/journal", 
              "dfs.namenode.path.based.cache.retry.interval.ms": "30000", 
              "dfs.namenode.inotify.max.events.per.rpc": "1000", 
              "dfs.datanode.cache.revocation.polling.ms": "500", 
              "mapreduce.reduce.skip.proc.count.autoincr": "true", 
              "file.replication": "1", 
              "mapreduce.jobhistory.cleaner.interval-ms": "86400000", 
              "hadoop.hdfs.configuration.version": "1", 
              "ipc.client.idlethreshold": "8000", 
              "yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore", 
              "hadoop.tmp.dir": "/tmp/hadoop-yarn-${hue.suffix}", 
              "mapreduce.jobtracker.restart.recover": "false", 
              "mapreduce.jobhistory.address": "host2:10020", 
              "mapreduce.cluster.local.dir": "/tmp/hadoop-yarn-${hue.suffix}/mapred/local", 
              "yarn.client.nodemanager-client-async.thread-pool-max-size": "500", 
              "proxyuser_group": "users", 
              "dfs.namenode.decommission.nodes.per.interval": "5", 
              "dfs.namenode.reject-unresolved-dn-topology-mapping": "false", 
              "yarn.nodemanager.resource.cpu-vcores": "20", 
              "dfs.namenode.delegation.key.update-interval": "86400000", 
              "dfs.client.read.shortcircuit.streams.cache.expiry.ms": "300000", 
              "fs.s3.buffer.dir": "/tmp/hadoop-yarn-${hue.suffix}/s3", 
              "dfs.namenode.support.allow.format": "true", 
              "yarn.nodemanager.remote-app-log-dir": "/app-logs", 
              "io.compression.codecs": "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec", 
              "yarn.nodemanager.aux-services": "mapreduce_shuffle", 
              "dfs.namenode.edit.log.autoroll.multiplier.threshold": "2.0", 
              "mapreduce.map.memory.mb": "3584", 
              "hadoop.work.around.non.threadsafe.getpwuid": "false", 
              "mapreduce.task.profile.reduce.params": "-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s", 
              "dfs.ha.automatic-failover.enabled": "true", 
              "yarn.timeline-service.client.max-retries": "30", 
              "dfs.namenode.stale.datanode.interval": "30000", 
              "dfs.namenode.edits.noeditlogchannelflush": "false", 
              "mapreduce.shuffle.transfer.buffer.size": "131072", 
              "dfs.namenode.logging.level": "info", 
              "mapreduce.jobtracker.persist.jobstatus.active": "true", 
              "yarn.nodemanager.log-dirs": "/var/log/hadoop-yarn", 
              "yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs": "86400", 
              "ha.health-monitor.sleep-after-disconnect.ms": "1000", 
              "dfs.namenode.checkpoint.edits.dir": "/grid/1/dfs/snn,/grid/2/dfs/snn,/grid/3/dfs/snn,/grid/4/dfs/snn,/grid/5/dfs/snn,/grid/6/dfs/snn", 
              "yarn.resourcemanager.fs.state-store.uri": " ", 
              "hadoop.rpc.socket.factory.class.default": "org.apache.hadoop.net.StandardSocketFactory", 
              "yarn.resourcemanager.keytab": "/etc/krb5.keytab", 
              "dfs.datanode.http.address": "0.0.0.0:50075", 
              "mapreduce.task.profile": "false", 
              "mapreduce.jobhistory.move.interval-ms": "180000", 
              "dfs.namenode.edits.dir": "/grid/1/dfs/nn,/grid/2/dfs/nn,/grid/3/dfs/nn,/grid/4/dfs/nn,/grid/5/dfs/nn,/grid/6/dfs/nn", 
              "dfs.storage.policy.enabled": "true", 
              "hadoop.security.kms.client.encrypted.key.cache.size": "500", 
              "hadoop.fuse.timer.period": "5", 
              "mapreduce.jobhistory.http.policy": "HTTP_ONLY", 
              "mapreduce.jobhistory.intermediate-done-dir": "/mr-history/tmp", 
              "mapreduce.map.skip.proc.count.autoincr": "true", 
              "dfs.cluster.administrators": " hdfs", 
              "fs.AbstractFileSystem.viewfs.impl": "org.apache.hadoop.fs.viewfs.ViewFs", 
              "mapreduce.job.speculative.slowtaskthreshold": "1.0", 
              "yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled": "false", 
              "s3native.stream-buffer-size": "4096", 
              "yarn.nodemanager.delete.debug-delay-sec": "0", 
              "dfs.secondary.namenode.kerberos.internal.spnego.principal": "${dfs.web.authentication.kerberos.principal}", 
              "dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold": "10737418240", 
              "fs.s3n.multipart.uploads.block.size": "67108864", 
              "mapreduce.ifile.readahead.bytes": "4194304", 
              "dfs.namenode.safemode.threshold-pct": "0.99f", 
              "yarn.scheduler.maximum-allocation-mb": "96768", 
              "ipc.client.fallback-to-simple-auth-allowed": "false", 
              "yarn.timeline-service.leveldb-timeline-store.read-cache-size": "104857600", 
              "fs.har.impl.disable.cache": "true", 
              "s3native.bytes-per-checksum": "512", 
              "yarn.timeline-service.hostname": "0.0.0.0", 
              "mapreduce.job.committer.setup.cleanup.needed": "true", 
              "fs.s3a.paging.maximum": "5000", 
              "yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms": "300000", 
              "yarn.client.nodemanager-connect.retry-interval-ms": "10000", 
              "yarn.nodemanager.log-aggregation.compression-type": "gz", 
              "yarn.app.mapreduce.am.job.committer.commit-window": "10000", 
              "hadoop.http.authentication.type": "simple", 
              "dfs.client.failover.sleep.base.millis": "500", 
              "dfs.ha.namenodes.$cluster_name": "nn1,nn2", 
              "yarn.nodemanager.vmem-check-enabled": "false", 
              "hadoop.jetty.logs.serve.aliases": "true", 
              "ha.failover-controller.graceful-fence.rpc-timeout.ms": "5000", 
              "mapreduce.reduce.shuffle.input.buffer.percent": "0.7", 
              "dfs.datanode.max.transfer.threads": "4096", 
              "mapreduce.task.io.sort.mb": "1024", 
              "mapreduce.reduce.merge.inmem.threshold": "1000", 
              "hadoop.security.kms.client.authentication.retry-count": "1", 
              "yarn.client.application-client-protocol.poll-interval-ms": "200", 
              "dfs.namenode.acls.enabled": "false", 
              "yarn.resourcemanager.connect.max-wait.ms": "900000", 
              "dfs.namenode.handler.count": "40", 
              "yarn.timeline-service.enabled": "true", 
              "dfs.namenode.retrycache.heap.percent": "0.03f", 
              "yarn.nodemanager.log.retain-second": "604800", 
              "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users": "true", 
              "yarn.resourcemanager.container.liveness-monitor.interval-ms": "600000", 
              "hadoop.ssl.client.conf": "ssl-client.xml", 
              "mapreduce.client.completion.pollinterval": "5000", 
              "yarn.nodemanager.vmem-pmem-ratio": "2.1", 
              "yarn.app.mapreduce.client.max-retries": "3", 
              "hadoop.ssl.enabled": "false", 
              "fs.AbstractFileSystem.hdfs.impl": "org.apache.hadoop.fs.Hdfs", 
              "fs.client.resolve.remote.symlinks": "true", 
              "mapreduce.reduce.java.opts": "-Xmx2867m", 
              "mapreduce.map.java.opts": "-Xmx2867m", 
              "mapreduce.tasktracker.reduce.tasks.maximum": "2", 
              "yarn.nodemanager.hostname": "0.0.0.0", 
              "mapreduce.reduce.input.buffer.percent": "0.0", 
              "fs.s3a.multipart.purge": "false", 
              "dfs.namenode.invalidate.work.pct.per.iteration": "0.32f", 
              "yarn.app.mapreduce.am.command-opts": "-Xmx2867m -Dhdp.version=2.2.0.0-2041", 
              "dfs.bytes-per-checksum": "512", 
              "dfs.replication": "3", 
              "yarn.resourcemanager.webapp.https.address": "host2:8090", 
              "mapreduce.shuffle.ssl.file.buffer.size": "65536", 
              "dfs.datanode.block.id.layout.upgrade.threads": "12", 
              "dfs.namenode.list.cache.directives.num.responses": "100", 
              "dfs.permissions.enabled": "true", 
              "mapreduce.jobtracker.maxtasks.perjob": "-1", 
              "dfs.datanode.use.datanode.hostname": "false", 
              "mapreduce.task.userlog.limit.kb": "0", 
              "dfs.namenode.fs-limits.max-directory-items": "1048576", 
              "s3.client-write-packet-size": "65536", 
              "fs.s3a.buffer.dir": "/tmp/hadoop-yarn-${hue.suffix}/s3a", 
              "hadoop.security.kms.client.encrypted.key.cache.low-watermark": "0.3f", 
              "hadoop.user.group.static.mapping.overrides": "dr.who=;", 
              "slider.provider.agent": "org.apache.slider.providers.agent.AgentProviderFactory", 
              "mapreduce.shuffle.max.threads": "0", 
              "dfs.client.failover.sleep.max.millis": "15000", 
              "hadoop.security.kms.client.encrypted.key.cache.expiry": "43200000", 
              "mapreduce.job.maps": "2", 
              "dfs.namenode.fs-limits.max-component-length": "255", 
              "hadoop.ssl.enabled.protocols": "TLSv1", 
              "mapreduce.map.output.compress": "false", 
              "s3.blocksize": "67108864", 
              "dfs.namenode.edits.journal-plugin.qjournal": "org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager", 
              "dfs.namenode.datanode.registration.ip-hostname-check": "true", 
              "yarn.node-labels.manager-class": "org.apache.hadoop.yarn.server.resourcemanager.nodelabels.MemoryRMNodeLabelsManager", 
              "yarn.nodemanager.pmem-check-enabled": "true", 
              "dfs.client.https.need-auth": "false", 
              "dfs.client.short.circuit.replica.stale.threshold.ms": "1800000", 
              "ha.zookeeper.quorum": "host1:2181,host2:2181,host3:2181", 
              "yarn.scheduler.minimum-allocation-mb": "3584", 
              "hadoop.proxyuser.hive.hosts": "host1", 
              "yarn.timeline-service.http-authentication.type": "simple", 
              "mapreduce.jobhistory.max-age-ms": "604800000", 
              "ftp.replication": "3", 
              "dfs.blockreport.split.threshold": "1000000", 
              "dfs.namenode.secondary.https-address": "0.0.0.0:50091", 
              "mapreduce.input.fileinputformat.split.minsize": "0", 
              "fs.s3n.block.size": "67108864", 
              "mapreduce.job.token.tracking.ids.enabled": "false", 
              "mapreduce.jobtracker.webinterface.trusted": "false", 
              "yarn.ipc.rpc.class": "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC", 
              "dfs.namenode.num.extra.edits.retained": "1000000", 
              "hadoop.http.staticuser.user": "dr.who", 
              "yarn.nodemanager.localizer.cache.cleanup.interval-ms": "600000", 
              "mapreduce.job.jvm.numtasks": "1", 
              "fs.s3a.multipart.size": "104857600", 
              "mapreduce.jobhistory.move.thread-count": "3", 
              "mapreduce.task.profile.maps": "0-2", 
              "dfs.datanode.max.locked.memory": "0", 
              "dfs.cachereport.intervalMsec": "10000", 
              "mapreduce.shuffle.port": "13562", 
              "mapreduce.shuffle.connection-keep-alive.timeout": "5", 
              "yarn.resourcemanager.nodemanager.minimum.version": "NONE", 
              "mapreduce.jobtracker.http.address": "0.0.0.0:50030", 
              "mapreduce.reduce.shuffle.merge.percent": "0.66", 
              "yarn.resourcemanager.connect.retry-interval.ms": "30000", 
              "mapreduce.task.skip.start.attempts": "2", 
              "yarn.scheduler.minimum-allocation-vcores": "1", 
              "mapreduce.task.io.sort.factor": "100", 
              "dfs.namenode.checkpoint.dir": "/grid/1/dfs/snn,/grid/2/dfs/snn,/grid/3/dfs/snn,/grid/4/dfs/snn,/grid/5/dfs/snn,/grid/6/dfs/snn", 
              "nfs.exports.allowed.hosts": "* rw", 
              "tfile.fs.input.buffer.size": "262144", 
              "tfile.io.chunk.size": "1048576", 
              "fs.s3.block.size": "67108864", 
              "fs.s3n.multipart.copy.block.size": "5368709120", 
              "io.serializations": "org.apache.hadoop.io.serializer.WritableSerialization", 
              "yarn.resourcemanager.max-completed-applications": "10000", 
              "mapreduce.jobhistory.principal": "jhs/_HOST@REALM.TLD", 
              "hadoop.proxyuser.hcat.hosts": "host1", 
              "yarn.resourcemanager.ha.automatic-failover.zk-base-path": "/yarn-leader-election", 
              "mapreduce.reduce.shuffle.fetch.retry.interval-ms": "1000", 
              "mapreduce.job.end-notification.retry.interval": "1000", 
              "mapreduce.application.framework.path": "/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework", 
              "dfs.namenode.backup.address": "0.0.0.0:50100", 
              "fs.s3n.multipart.uploads.enabled": "false", 
              "s3native.client-write-packet-size": "65536", 
              "dfs.block.access.token.enable": "true", 
              "io.seqfile.sorter.recordlimit": "1000000", 
              "hadoop.security.group.mapping": "org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback", 
              "ftp.bytes-per-checksum": "512", 
              "dfs.namenode.fs-limits.max-xattr-size": "16384", 
              "dfs.client.read.shortcircuit.streams.cache.size": "4096", 
              "dfs.client.domain.socket.data.traffic": "false", 
              "fs.s3a.connection.timeout": "5000", 
              "mapreduce.job.end-notification.max.retry.interval": "5000", 
              "dfs.namenode.rpc-address.$cluster_name.nn1": "host1:8020", 
              "yarn.acl.enable": "false", 
              "mapreduce.application.classpath": "$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.2.0.0-2041/hadoop/lib/hadoop-lzo-0.6.0.2.2.0.0-2041.jar:/etc/hadoop/conf/secure", 
              "yarn.nm.liveness-monitor.expiry-interval-ms": "600000", 
              "dfs.namenode.rpc-address.$cluster_name.nn2": "host2:8020", 
              "dfs.client.mmap.cache.size": "256", 
              "mapreduce.input.fileinputformat.list-status.num-threads": "1", 
              "mapreduce.tasktracker.map.tasks.maximum": "2", 
              "yarn.nodemanager.linux-container-executor.resources-handler.class": "org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler", 
              "yarn.timeline-service.ttl-enable": "true", 
              "yarn.resourcemanager.zk-address": "localhost:2181", 
              "dfs.namenode.max.objects": "0", 
              "yarn.resourcemanager.state-store.max-completed-applications": "10000", 
              "dfs.namenode.delegation.token.max-lifetime": "604800000", 
              "mapreduce.job.classloader": "false", 
              "yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size": "10000", 
              "mapreduce.job.hdfs-servers": "hdfs://$cluster_name", 
              "yarn.application.classpath": "$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*", 
              "mapreduce.tasktracker.dns.nameserver": "default", 
              "dfs.datanode.hdfs-blocks-metadata.enabled": "false", 
              "dfs.datanode.readahead.bytes": "4193404", 
              "dfs.image.compress": "false", 
              "mapreduce.job.ubertask.maxreduces": "1", 
              "mapreduce.tasktracker.report.address": "127.0.0.1:0", 
              "yarn.log-aggregation-enable": "true", 
              "mapreduce.shuffle.ssl.enabled": "false", 
              "rpc.engine.org.apache.slider.server.appmaster.rpc.SliderClusterProtocolPB": "org.apache.hadoop.ipc.ProtobufRpcEngine", 
              "mapreduce.tasktracker.http.threads": "40", 
              "dfs.stream-buffer-size": "4096", 
              "tfile.fs.output.buffer.size": "262144", 
              "fs.permissions.umask-mode": "022", 
              "yarn.resourcemanager.am.max-attempts": "2", 
              "dfs.namenode.resource.du.reserved": "104857600", 
              "dfs.client.datanode-restart.timeout": "30", 
              "yarn.nodemanager.resource.percentage-physical-cpu-limit": "100", 
              "ha.failover-controller.graceful-fence.connection.retries": "1", 
              "dfs.datanode.drop.cache.behind.writes": "false", 
              "yarn.app.mapreduce.am.resource.cpu-vcores": "1", 
              "mapreduce.job.ubertask.enable": "false", 
              "hadoop.common.configuration.version": "0.23.0", 
              "dfs.namenode.replication.work.multiplier.per.iteration": "2", 
              "mapreduce.job.acl-modify-job": " ", 
              "io.seqfile.local.dir": "/tmp/hadoop-yarn-${hue.suffix}/io/local", 
              "fs.s3.sleepTimeSeconds": "10", 
              "yarn.resourcemanager.system-metrics-publisher.enabled": "true", 
              "mapreduce.client.output.filter": "FAILED"
          }, 
          "empty": false
      }
      
      1. app0001.zip
        1.05 MB
        Gour Saha
      2. application_1431771946377_0001.zip
        1.36 MB
        Chackaravarthy

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Duplicate of HADOOP-12317.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Duplicate of HADOOP-12317 .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          I see you filed HADOOP-11989. Assuming that is the root-cause, we can close this as a duplicate.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - I see you filed HADOOP-11989 . Assuming that is the root-cause, we can close this as a duplicate.
          Hide
          chackra Chackaravarthy added a comment -

          Improper (specific to this env) kill command construction was the issue. I tested by making changes in Shell.java class to construct the kill command as follows (including two hyphens) :

          kill -signalNo -- -<process_id>
          

          It works fine with this change in debian7.

          Show
          chackra Chackaravarthy added a comment - Improper (specific to this env) kill command construction was the issue. I tested by making changes in Shell.java class to construct the kill command as follows (including two hyphens) : kill -signalNo -- -<process_id> It works fine with this change in debian7.
          Hide
          chackra Chackaravarthy added a comment -

          Seems like the issue is related with 'setsid' enabled in the hosts and unable to execute the 'kill' command constructed (with hyphen).

          Analysis as follows,

          From debug logs, its found that SIGTERM signal sent to SliderAgent by NM did not succeed.

          2015-05-16 15:59:08,588 DEBUG launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(411)) - Sending signal to pid 4083 as user yarn for container container_1431771946377_0001_01_000002
          2015-05-16 15:59:08,589 DEBUG nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:signalContainer(399)) - Sending signal 15 to pid 4083 as user yarn
          2015-05-16 15:59:08,596 DEBUG launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(421)) - Sent signal SIGTERM to pid 4083 as user yarn for container container_1431771946377_0001_01_000002, result=failed
          2015-05-16 15:59:08,848 DEBUG nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:signalContainer(399)) - Sending signal 9 to pid 4083 as user yarn

          NM gets the PID for the running slider agent and tries to send SIGTERM(15) first, but it fails. Then it tries to send SIGKILL(9) and that too fails.
          The reason being containerIsAlive(pid) method always returning false and hence killContainer() method never executed which need to send KILL signal.

          DefaultContainerExecutor.java
          // 
          public boolean signalContainer(String user, String pid, Signal signal)
          {
              if (!containerIsAlive(pid)) {
                return false;
              }
          	killContainer(pid, signal) // sending kill signal here only.... AND IT NEVER REACHES HERE
              return true;
          }
          

          containerIsAlive returning false because of improper construction of kill command which results in ExitCodeException :

          kill -0 -<proc_id>

          kill -0 -4083

          Shell.java
          /** Return a command for determining if process with specified pid is alive. */
            public static String[] getCheckProcessIsAliveCommand(String pid) {
              return Shell.WINDOWS ?
                new String[] { Shell.WINUTILS, "task", "isAlive", pid } :
                new String[] { "kill", "-0", isSetsidAvailable ? "-" + pid : pid };
            }
          

          In this environment, setsid been enabled and hence "-" been added as prefix while constructing this command. But not sure why this command execution gets ExitCodeException, if this is the correct way of executing command in setsid enabled environment. When I try to execute the same command (with hyphen) in the host shell, same improper usage error came.

          Show
          chackra Chackaravarthy added a comment - Seems like the issue is related with 'setsid' enabled in the hosts and unable to execute the 'kill' command constructed (with hyphen). Analysis as follows, From debug logs, its found that SIGTERM signal sent to SliderAgent by NM did not succeed. 2015-05-16 15:59:08,588 DEBUG launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(411)) - Sending signal to pid 4083 as user yarn for container container_1431771946377_0001_01_000002 2015-05-16 15:59:08,589 DEBUG nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:signalContainer(399)) - Sending signal 15 to pid 4083 as user yarn 2015-05-16 15:59:08,596 DEBUG launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(421)) - Sent signal SIGTERM to pid 4083 as user yarn for container container_1431771946377_0001_01_000002, result=failed 2015-05-16 15:59:08,848 DEBUG nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:signalContainer(399)) - Sending signal 9 to pid 4083 as user yarn NM gets the PID for the running slider agent and tries to send SIGTERM(15) first, but it fails. Then it tries to send SIGKILL(9) and that too fails. The reason being containerIsAlive(pid) method always returning false and hence killContainer() method never executed which need to send KILL signal. DefaultContainerExecutor.java // public boolean signalContainer( String user, String pid, Signal signal) { if (!containerIsAlive(pid)) { return false ; } killContainer(pid, signal) // sending kill signal here only.... AND IT NEVER REACHES HERE return true ; } containerIsAlive returning false because of improper construction of kill command which results in ExitCodeException : kill -0 -<proc_id> kill -0 -4083 Shell.java /** Return a command for determining if process with specified pid is alive. */ public static String [] getCheckProcessIsAliveCommand( String pid) { return Shell.WINDOWS ? new String [] { Shell.WINUTILS, "task" , "isAlive" , pid } : new String [] { "kill" , "-0" , isSetsidAvailable ? "-" + pid : pid }; } In this environment, setsid been enabled and hence "-" been added as prefix while constructing this command. But not sure why this command execution gets ExitCodeException, if this is the correct way of executing command in setsid enabled environment. When I try to execute the same command (with hyphen) in the host shell, same improper usage error came.
          Hide
          chackra Chackaravarthy added a comment -

          Gour Saha / Jian He

          Attached logs (application_1431771946377_0001.zip) with debug level enabled. It contains RM and NM logs from hosts running Slider AM and non-AM application containers.

          container_1431771946377_0001_01_000001 - host3 - SliderAM
          container_1431771946377_0001_01_000002 - host7 - NIMBUS
          container_1431771946377_0001_01_000003 - host5 - STORM_UI_SERVER
          container_1431771946377_0001_01_000004 - host3 - DRPC_SERVER
          container_1431771946377_0001_01_000005 - host6 - SUPERVISOR

          Timing of issuing the commands:

          Slider start command : 2015-05-16 15:57:11,954
          Slider stop command : 2015-05-16 15:59:06,480

          Show
          chackra Chackaravarthy added a comment - Gour Saha / Jian He Attached logs (application_1431771946377_0001.zip) with debug level enabled. It contains RM and NM logs from hosts running Slider AM and non-AM application containers. container_1431771946377_0001_01_000001 - host3 - SliderAM container_1431771946377_0001_01_000002 - host7 - NIMBUS container_1431771946377_0001_01_000003 - host5 - STORM_UI_SERVER container_1431771946377_0001_01_000004 - host3 - DRPC_SERVER container_1431771946377_0001_01_000005 - host6 - SUPERVISOR Timing of issuing the commands: Slider start command : 2015-05-16 15:57:11,954 Slider stop command : 2015-05-16 15:59:06,480
          Hide
          gsaha Gour Saha added a comment -

          Jian He it is consistently reproducible on debian 7. Can you provide a quick instruction on how to enable debug level in NM logs?

          Chackaravarthy If possible, can you set debug level on for NM logs, re-run the test and provide the logs again?

          Show
          gsaha Gour Saha added a comment - Jian He it is consistently reproducible on debian 7. Can you provide a quick instruction on how to enable debug level in NM logs? Chackaravarthy If possible, can you set debug level on for NM logs, re-run the test and provide the logs again?
          Hide
          jianhe Jian He added a comment -

          Gour Saha, from the description, this is running against 2.6 ? this could be related to YARN-2825, but that's fixed in 2.6
          From the logs, I can only see the container is still sort of waiting for the process to finish. Is this easy to reproduce? It'll be great if we have NM logs with debug level on.

          Show
          jianhe Jian He added a comment - Gour Saha , from the description, this is running against 2.6 ? this could be related to YARN-2825 , but that's fixed in 2.6 From the logs, I can only see the container is still sort of waiting for the process to finish. Is this easy to reproduce? It'll be great if we have NM logs with debug level on.
          Hide
          gsaha Gour Saha added a comment -

          Jian He The logs which were used to file the bug was not available anymore. Hence logs for a newer run were uploaded as a zip. Please look into container with id container_1430849660771_0001_01_000002.

          Show
          gsaha Gour Saha added a comment - Jian He The logs which were used to file the bug was not available anymore. Hence logs for a newer run were uploaded as a zip. Please look into container with id container_1430849660771_0001_01_000002 .
          Hide
          jianhe Jian He added a comment -

          Gour Saha, I tried to look at this, but couldn't find the "container_1428575950531_0021_01_000002" log in any of the uploaded NM logs. could you point me which container is misbehaving in any of the NM logs you uploaded here ?

          Show
          jianhe Jian He added a comment - Gour Saha , I tried to look at this, but couldn't find the "container_1428575950531_0021_01_000002" log in any of the uploaded NM logs. could you point me which container is misbehaving in any of the NM logs you uploaded here ?
          Hide
          gsaha Gour Saha added a comment -

          Logs for the run used to file the bug are not available anymore.

          Attached logs (app0001.zip) for one of the recent attempts. It contains RM and NM logs from hosts running Slider AM and non-AM application containers.

          container_1430849660771_0001_01_000001 - host04 - SliderAM
          container_1430849660771_0001_01_000002 - host04 - NIMBUS
          container_1430849660771_0001_01_000003 - host05 - STORM_UI_SERVER
          container_1430849660771_0001_01_000004 - host03 - DRPC_SERVER
          container_1430849660771_0001_01_000005 - host07 - SUPERVISOR

          Timing of issuing the commands:
          Slider start command : 2015-05-06 00:01:04,710
          Slider stop command : 2015-05-06 00:03:13,608

          Show
          gsaha Gour Saha added a comment - Logs for the run used to file the bug are not available anymore. Attached logs (app0001.zip) for one of the recent attempts. It contains RM and NM logs from hosts running Slider AM and non-AM application containers. container_1430849660771_0001_01_000001 - host04 - SliderAM container_1430849660771_0001_01_000002 - host04 - NIMBUS container_1430849660771_0001_01_000003 - host05 - STORM_UI_SERVER container_1430849660771_0001_01_000004 - host03 - DRPC_SERVER container_1430849660771_0001_01_000005 - host07 - SUPERVISOR Timing of issuing the commands: Slider start command : 2015-05-06 00:01:04,710 Slider stop command : 2015-05-06 00:03:13,608
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Could this be OS specific (debian 7)?

          Possible. Can you post the full NM logs?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Could this be OS specific (debian 7)? Possible. Can you post the full NM logs?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Rakesh Kumar Sahoo (accidentally?) made it patch-available. Canceling it..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Rakesh Kumar Sahoo (accidentally?) made it patch-available. Canceling it..
          Hide
          gsaha Gour Saha added a comment -

          Wangda Tan thanks for linking YARN-2904. Seems like there is something common between the two.

          Slider non-AM containers register a SIGINT and SIGTERM at the start. When Slider AM initiates a graceful shutdown, all non-AM containers receive a SIGTERM.

          In this scenario, I saw that the non-AM containers never received a SIGTERM.

          I checked /proc/<pid>/status in one of those nodes and the signals SIGINT and SIGTERM were registered correctly

          SigCgt: 0000000180004202
          
          Show
          gsaha Gour Saha added a comment - Wangda Tan thanks for linking YARN-2904 . Seems like there is something common between the two. Slider non-AM containers register a SIGINT and SIGTERM at the start. When Slider AM initiates a graceful shutdown, all non-AM containers receive a SIGTERM. In this scenario, I saw that the non-AM containers never received a SIGTERM. I checked /proc/<pid>/status in one of those nodes and the signals SIGINT and SIGTERM were registered correctly SigCgt: 0000000180004202
          Hide
          gsaha Gour Saha added a comment -

          I checked the code and Slider does set keepContainersAcrossApplicationAttempts to true.

          In this case a graceful Slider stop was issued, which calls AMRMClientAsync.releaseAssignedContainer for all containers. The NodeManager also initiated the stop of all containers. As seen below in the log snippet, the container state transitioned from RUNNING to KILLING and the application state transitioned from RUNNING to FINISHING_CONTAINERS_WAIT, but got lost somewhere and the resource-monitor continued to run.

          Also note, that we never encountered this issue in our test and dev environments. Could this be OS specific (debian 7)? The Slider AM log snippet from the time it received the stop command is provided below.

          Snippet from NM log in host-03

          2015-04-29 00:56:00,486 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(953)) - Stopping container with container Id: container_1428575950531_0021_01_000002
          2015-04-29 00:56:00,487 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn	IP=10.84.104.129	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1428575950531_0021	CONTAINERID=container_1428575950531_0021_01_000002
          2015-04-29 00:56:00,487 INFO  container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from RUNNING to KILLING
          2015-04-29 00:56:00,489 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container container_1428575950531_0021_01_000002
          2015-04-29 00:56:00,802 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
          2015-04-29 00:56:02,494 INFO  application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
          2015-04-29 00:56:03,849 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
          2015-04-29 00:56:06,892 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used
          .
          .
          .
          

          Slider AM log:

          2015-04-29 00:55:59,189 [IPC Server handler 0 on 1024] INFO  appmaster.SliderAppMaster - SliderAppMasterApi.stopCluster: stop command issued:  exit code = 0, SUCCEEDED: stop command issued;
          2015-04-29 00:56:00,189 [AmExecutor-006] INFO  appmaster.SliderAppMaster - SliderAppMasterApi.stopCluster: stop command issued
          2015-04-29 00:56:00,190 [main] INFO  appmaster.SliderAppMaster - Triggering shutdown of the AM: stop command issued:  exit code = 0, SUCCEEDED: stop command issued;
          2015-04-29 00:56:00,190 [main] INFO  appmaster.SliderAppMaster - Process has exited with exit code 0 mapped to 0 -ignoring
          2015-04-29 00:56:00,190 [main] INFO  workflow.WorkflowCompositeService - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED
          2015-04-29 00:56:00,191 [main] INFO  state.AppState - Releasing 5 containers
          2015-04-29 00:56:00,191 [main] INFO  state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-05:45454/container_1428575950531_0021_01_000003/ctx/yarn
          2015-04-29 00:56:00,192 [main] INFO  state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-07:45454/container_1428575950531_0021_01_000005/ctx/yarn
          2015-04-29 00:56:00,192 [main] INFO  state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-03:45454/container_1428575950531_0021_01_000002/ctx/yarn
          2015-04-29 00:56:00,193 [main] INFO  state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-04:45454/container_1428575950531_0021_01_000004/ctx/yarn
          2015-04-29 00:56:00,193 [main] INFO  appmaster.SliderAppMaster - Application completed. Signalling finish to RM
          2015-04-29 00:56:00,193 [main] INFO  appmaster.SliderAppMaster - Unregistering AM status=SUCCEEDED message=stop command issued
          2015-04-29 00:56:00,202 [main] INFO  impl.AMRMClientImpl - Waiting for application to be successfully unregistered.
          2015-04-29 00:56:00,304 [main] INFO  appmaster.SliderAppMaster - Exiting AM; final exit code = 0
          2015-04-29 00:56:00,306 [main] INFO  util.ExitUtil - Exiting with status 0
          2015-04-29 00:56:00,307 [Shutdown] INFO  mortbay.log - Shutdown hook executing
          2015-04-29 00:56:00,307 [Shutdown] INFO  mortbay.log - Stopped SslSelectChannelConnector@0.0.0.0:54797
          2015-04-29 00:56:00,311 [Shutdown] INFO  mortbay.log - Stopped SslSelectChannelConnector@0.0.0.0:41739
          2015-04-29 00:56:00,315 [Thread-1] INFO  mortbay.log - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1025
          2015-04-29 00:56:00,414 [Shutdown] INFO  mortbay.log - Shutdown hook complete
          2015-04-29 00:56:00,425 [Thread-1] INFO  ipc.Server - Stopping server on 1024
          2015-04-29 00:56:00,426 [IPC Server listener on 1024] INFO  ipc.Server - Stopping IPC Server listener on 1024
          2015-04-29 00:56:00,427 [IPC Server Responder] INFO  ipc.Server - Stopping IPC Server Responder
          2015-04-29 00:56:00,429 [Thread-1] INFO  impl.ContainerManagementProtocolProxy - Opening proxy : host-05:45454
          2015-04-29 00:56:00,455 [Thread-1] INFO  impl.ContainerManagementProtocolProxy - Opening proxy : host-07:45454
          2015-04-29 00:56:00,470 [Thread-1] INFO  impl.ContainerManagementProtocolProxy - Opening proxy : host-03:45454
          2015-04-29 00:56:00,491 [Thread-1] INFO  impl.ContainerManagementProtocolProxy - Opening proxy : host-04:45454
          2015-04-29 00:56:00,507 [AMRM Callback Handler Thread] INFO  impl.AMRMClientAsyncImpl - Interrupted while waiting for queue
          java.lang.InterruptedException
                  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
                  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
                  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
                  at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
          2015-04-29 00:56:00,507 [AmExecutor-005] INFO  actions.QueueService - QueueService processor terminated
          2015-04-29 00:56:00,507 [AmExecutor-006] WARN  actions.ActionStopQueue - STOP
          
          Show
          gsaha Gour Saha added a comment - I checked the code and Slider does set keepContainersAcrossApplicationAttempts to true. In this case a graceful Slider stop was issued, which calls AMRMClientAsync.releaseAssignedContainer for all containers. The NodeManager also initiated the stop of all containers. As seen below in the log snippet, the container state transitioned from RUNNING to KILLING and the application state transitioned from RUNNING to FINISHING_CONTAINERS_WAIT , but got lost somewhere and the resource-monitor continued to run. Also note, that we never encountered this issue in our test and dev environments. Could this be OS specific (debian 7)? The Slider AM log snippet from the time it received the stop command is provided below. Snippet from NM log in host-03 2015-04-29 00:56:00,486 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(953)) - Stopping container with container Id: container_1428575950531_0021_01_000002 2015-04-29 00:56:00,487 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=yarn IP=10.84.104.129 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1428575950531_0021 CONTAINERID=container_1428575950531_0021_01_000002 2015-04-29 00:56:00,487 INFO container.Container (ContainerImpl.java:handle(999)) - Container container_1428575950531_0021_01_000002 transitioned from RUNNING to KILLING 2015-04-29 00:56:00,489 INFO launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container container_1428575950531_0021_01_000002 2015-04-29 00:56:00,802 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.4 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used 2015-04-29 00:56:02,494 INFO application.Application (ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT 2015-04-29 00:56:03,849 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used 2015-04-29 00:56:06,892 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 21259 for container-id container_1428575950531_0021_01_000002: 15.5 MB of 3.5 GB physical memory used; 280.3 MB of 7.3 GB virtual memory used . . . Slider AM log: 2015-04-29 00:55:59,189 [IPC Server handler 0 on 1024] INFO appmaster.SliderAppMaster - SliderAppMasterApi.stopCluster: stop command issued: exit code = 0, SUCCEEDED: stop command issued; 2015-04-29 00:56:00,189 [AmExecutor-006] INFO appmaster.SliderAppMaster - SliderAppMasterApi.stopCluster: stop command issued 2015-04-29 00:56:00,190 [main] INFO appmaster.SliderAppMaster - Triggering shutdown of the AM: stop command issued: exit code = 0, SUCCEEDED: stop command issued; 2015-04-29 00:56:00,190 [main] INFO appmaster.SliderAppMaster - Process has exited with exit code 0 mapped to 0 -ignoring 2015-04-29 00:56:00,190 [main] INFO workflow.WorkflowCompositeService - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED 2015-04-29 00:56:00,191 [main] INFO state.AppState - Releasing 5 containers 2015-04-29 00:56:00,191 [main] INFO state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-05:45454/container_1428575950531_0021_01_000003/ctx/yarn 2015-04-29 00:56:00,192 [main] INFO state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-07:45454/container_1428575950531_0021_01_000005/ctx/yarn 2015-04-29 00:56:00,192 [main] INFO state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-03:45454/container_1428575950531_0021_01_000002/ctx/yarn 2015-04-29 00:56:00,193 [main] INFO state.AppState - Releasing container. Log: http://host-02:19888/jobhistory/logs/host-04:45454/container_1428575950531_0021_01_000004/ctx/yarn 2015-04-29 00:56:00,193 [main] INFO appmaster.SliderAppMaster - Application completed. Signalling finish to RM 2015-04-29 00:56:00,193 [main] INFO appmaster.SliderAppMaster - Unregistering AM status=SUCCEEDED message=stop command issued 2015-04-29 00:56:00,202 [main] INFO impl.AMRMClientImpl - Waiting for application to be successfully unregistered. 2015-04-29 00:56:00,304 [main] INFO appmaster.SliderAppMaster - Exiting AM; final exit code = 0 2015-04-29 00:56:00,306 [main] INFO util.ExitUtil - Exiting with status 0 2015-04-29 00:56:00,307 [Shutdown] INFO mortbay.log - Shutdown hook executing 2015-04-29 00:56:00,307 [Shutdown] INFO mortbay.log - Stopped SslSelectChannelConnector@0.0.0.0:54797 2015-04-29 00:56:00,311 [Shutdown] INFO mortbay.log - Stopped SslSelectChannelConnector@0.0.0.0:41739 2015-04-29 00:56:00,315 [Thread-1] INFO mortbay.log - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:1025 2015-04-29 00:56:00,414 [Shutdown] INFO mortbay.log - Shutdown hook complete 2015-04-29 00:56:00,425 [Thread-1] INFO ipc.Server - Stopping server on 1024 2015-04-29 00:56:00,426 [IPC Server listener on 1024] INFO ipc.Server - Stopping IPC Server listener on 1024 2015-04-29 00:56:00,427 [IPC Server Responder] INFO ipc.Server - Stopping IPC Server Responder 2015-04-29 00:56:00,429 [Thread-1] INFO impl.ContainerManagementProtocolProxy - Opening proxy : host-05:45454 2015-04-29 00:56:00,455 [Thread-1] INFO impl.ContainerManagementProtocolProxy - Opening proxy : host-07:45454 2015-04-29 00:56:00,470 [Thread-1] INFO impl.ContainerManagementProtocolProxy - Opening proxy : host-03:45454 2015-04-29 00:56:00,491 [Thread-1] INFO impl.ContainerManagementProtocolProxy - Opening proxy : host-04:45454 2015-04-29 00:56:00,507 [AMRM Callback Handler Thread] INFO impl.AMRMClientAsyncImpl - Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) 2015-04-29 00:56:00,507 [AmExecutor-005] INFO actions.QueueService - QueueService processor terminated 2015-04-29 00:56:00,507 [AmExecutor-006] WARN actions.ActionStopQueue - STOP
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Vinod probably means the AM restart flag.

          When the Slider AM is stopped it can be done in two ways

          slider stop $clustername
          

          Sends an RPC call to the AM, which then unregisters and shuts down. I'll check to make sure we explicitly release containers.

          slider stop $clustername --force
          

          This asks YARN to kill the app; the AM doesn't get told about it.

          there's a third way

          slider am-suicide $clustername
          

          This is only for testing, causes the AM to call System.exit(-1); YARN will restart it unless it has failed too many times already.

          Show
          stevel@apache.org Steve Loughran added a comment - Vinod probably means the AM restart flag. When the Slider AM is stopped it can be done in two ways slider stop $clustername Sends an RPC call to the AM, which then unregisters and shuts down. I'll check to make sure we explicitly release containers. slider stop $clustername --force This asks YARN to kill the app; the AM doesn't get told about it. there's a third way slider am-suicide $clustername This is only for testing, causes the AM to call System.exit(-1) ; YARN will restart it unless it has failed too many times already.
          Hide
          gsaha Gour Saha added a comment -

          Slider stop command was called which initiates the Slider Storm application to stop (and hence the Slider AM to stop).

          Which property sets the keep-containers flag on?

          Show
          gsaha Gour Saha added a comment - Slider stop command was called which initiates the Slider Storm application to stop (and hence the Slider AM to stop). Which property sets the keep-containers flag on?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Is this because the keep-containers flag is on? Why was the AM stopped and not the the app killed if that is what they want.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Is this because the keep-containers flag is on? Why was the AM stopped and not the the app killed if that is what they want.

            People

            • Assignee:
              Unassigned
              Reporter:
              gsaha Gour Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development