2018-04-20 16:44:14,983 [main] INFO service.ServiceMaster - STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting ServiceMaster STARTUP_MSG: host = eyang-5.openstacklocal/172.26.111.20 STARTUP_MSG: args = [-yarnfile, hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/abc.json, -D, hadoop.registry.zk.root=/registry, -D, hadoop.registry.zk.quorum=eyang-1.openstacklocal:2181] STARTUP_MSG: version = 3.2.0-SNAPSHOT STARTUP_MSG: classpath = yarnservice-log4j.properties:conf/:lib/jaxb-api-2.2.11.jar:lib/hadoop-yarn-server-timeline-pluginstorage-3.2.0-SNAPSHOT.jar:lib/jetty-util-ajax-9.3.19.v20170502.jar:lib/hadoop-yarn-applications-distributedshell-3.2.0-SNAPSHOT.jar:lib/kerby-util-1.0.1.jar:lib/okio-1.6.0.jar:lib/commons-io-2.5.jar:lib/kerb-simplekdc-1.0.1.jar:lib/hadoop-yarn-api-3.2.0-SNAPSHOT.jar:lib/commons-compress-1.4.1.jar:lib/hadoop-hdfs-rbf-3.2.0-SNAPSHOT.jar:lib/fst-2.50.jar:lib/nimbus-jose-jwt-4.41.1.jar:lib/junit-4.11.jar:lib/snakeyaml-1.16.jar:lib/yarn-service-core.jar:lib/jsch-0.1.54.jar:lib/kerb-identity-1.0.1.jar:lib/hadoop-yarn-server-resourcemanager-3.2.0-SNAPSHOT.jar:lib/kerby-asn1-1.0.1.jar:lib/dnsjava-2.1.7.jar:lib/hadoop-yarn-services-core-3.2.0-SNAPSHOT.jar:lib/token-provider-1.0.1.jar:lib/snappy-java-1.0.5.jar:lib/hadoop-common-3.2.0-SNAPSHOT.jar:lib/hadoop-yarn-registry-3.2.0-SNAPSHOT.jar:lib/zookeeper-3.4.9.jar:lib/jersey-server-1.19.jar:lib/kerb-admin-1.0.1.jar:lib/jackson-core-asl-1.9.13.jar:lib/kerb-crypto-1.0.1.jar:lib/jetty-webapp-9.3.19.v20170502.jar:lib/hadoop-yarn-services-api-3.2.0-SNAPSHOT.jar:lib/mssql-jdbc-6.2.1.jre7.jar:lib/curator-framework-2.12.0.jar:lib/gson-2.2.4.jar:lib/jetty-servlet-9.3.19.v20170502.jar:lib/xz-1.0.jar:lib/hadoop-common-3.2.0-SNAPSHOT-tests.jar:lib/hadoop-yarn-applications-unmanaged-am-launcher-3.2.0-SNAPSHOT.jar:lib/commons-lang3-3.7.jar:lib/okhttp-2.7.5.jar:lib/commons-math3-3.1.1.jar:lib/jackson-xc-1.9.13.jar:lib/hadoop-kms-3.2.0-SNAPSHOT.jar:lib/jersey-json-1.19.jar:lib/kerb-server-1.0.1.jar:lib/kerb-client-1.0.1.jar:lib/hadoop-yarn-server-router-3.2.0-SNAPSHOT.jar:lib/json-io-2.5.1.jar:lib/metrics-core-3.2.4.jar:lib/curator-recipes-2.12.0.jar:lib/javax.servlet-api-3.1.0.jar:lib/hadoop-yarn-server-common-3.2.0-SNAPSHOT.jar:lib/hadoop-hdfs-rbf-3.2.0-SNAPSHOT-tests.jar:lib/jcip-annotations-1.0-1.jar:lib/jetty-xml-9.3.19.v20170502.jar:lib/re2j-1.1.jar:lib/asm-5.0.4.jar:lib/kerb-util-1.0.1.jar:lib/protobuf-java-2.5.0.jar:lib/hadoop-hdfs-nfs-3.2.0-SNAPSHOT.jar:lib/kerb-common-1.0.1.jar:lib/swagger-annotations-1.5.4.jar:lib/commons-net-3.6.jar:lib/commons-cli-1.2.jar:lib/slf4j-api-1.7.25.jar:lib/jsr305-3.0.0.jar:lib/java-util-1.9.0.jar:lib/aopalliance-1.0.jar:lib/kerby-pkix-1.0.1.jar:lib/jettison-1.1.jar:lib/slf4j-log4j12-1.7.25.jar:lib/httpcore-4.4.4.jar:lib/hadoop-yarn-common-3.2.0-SNAPSHOT.jar:lib/hadoop-hdfs-3.2.0-SNAPSHOT-tests.jar:lib/hadoop-yarn-server-tests-3.2.0-SNAPSHOT.jar:lib/jul-to-slf4j-1.7.25.jar:lib/jetty-server-9.3.19.v20170502.jar:lib/HikariCP-java7-2.4.12.jar:lib/kerby-config-1.0.1.jar:lib/jackson-jaxrs-json-provider-2.9.4.jar:lib/htrace-core4-4.1.0-incubating.jar:lib/hadoop-hdfs-3.2.0-SNAPSHOT.jar:lib/netty-all-4.0.52.Final.jar:lib/ehcache-3.3.1.jar:lib/curator-client-2.12.0.jar:lib/jersey-servlet-1.19.jar:lib/jersey-client-1.19.jar:lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:lib/commons-daemon-1.0.13.jar:lib/commons-codec-1.11.jar:lib/guice-servlet-4.0.jar:lib/jsp-api-2.1.jar:lib/hadoop-yarn-server-sharedcachemanager-3.2.0-SNAPSHOT.jar:lib/hadoop-hdfs-native-client-3.2.0-SNAPSHOT.jar:lib/hadoop-annotations-3.2.0-SNAPSHOT.jar:lib/avro-1.7.7.jar:lib/guice-4.0.jar:lib/hadoop-yarn-server-applicationhistoryservice-3.2.0-SNAPSHOT.jar:lib/commons-configuration2-2.1.1.jar:lib/hadoop-yarn-client-3.2.0-SNAPSHOT.jar:lib/hadoop-hdfs-native-client-3.2.0-SNAPSHOT-tests.jar:lib/jackson-jaxrs-base-2.9.4.jar:lib/kerb-core-1.0.1.jar:lib/commons-beanutils-1.9.3.jar:lib/netty-3.10.5.Final.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/jersey-guice-1.19.jar:lib/guava-11.0.2.jar:lib/jetty-util-9.3.19.v20170502.jar:lib/hadoop-nfs-3.2.0-SNAPSHOT.jar:lib/jersey-core-1.19.jar:lib/kerby-xdr-1.0.1.jar:lib/jsr311-api-1.1.1.jar:lib/stax2-api-3.1.4.jar:lib/httpclient-4.5.2.jar:lib/json-simple-1.1.1.jar:lib/hadoop-auth-3.2.0-SNAPSHOT.jar:lib/hadoop-yarn-server-web-proxy-3.2.0-SNAPSHOT.jar:lib/accessors-smart-1.2.jar:lib/hamcrest-core-1.3.jar:lib/commons-collections-3.2.2.jar:lib/leveldbjni-all-1.8.jar:lib/jetty-security-9.3.19.v20170502.jar:lib/jackson-databind-2.9.4.jar:lib/commons-logging-1.1.3.jar:lib/hadoop-hdfs-client-3.2.0-SNAPSHOT.jar:lib/jackson-jaxrs-1.9.13.jar:lib/hadoop-hdfs-httpfs-3.2.0-SNAPSHOT.jar:lib/jackson-module-jaxb-annotations-2.9.4.jar:lib/jaxb-impl-2.2.3-1.jar:lib/jackson-annotations-2.9.4.jar:lib/commons-lang-2.6.jar:lib/hadoop-yarn-server-nodemanager-3.2.0-SNAPSHOT.jar:lib/mockito-all-1.8.5.jar:lib/jetty-io-9.3.19.v20170502.jar:lib/javax.inject-1.jar:lib/log4j-1.2.17.jar:lib/jetty-http-9.3.19.v20170502.jar:lib/json-smart-2.3.jar:lib/jackson-core-2.9.4.jar:lib/hadoop-hdfs-client-3.2.0-SNAPSHOT-tests.jar:lib/paranamer-2.3.jar:lib/woodstox-core-5.0.3.jar::/usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop STARTUP_MSG: build = https://github.com/apache/hadoop -r c6d7d3eb059c7539db7d00586e181ec44da13557; compiled by 'eyang' on 2018-04-20T15:40Z STARTUP_MSG: java = 1.8.0_141 ************************************************************/ 2018-04-20 16:44:14,993 [main] INFO service.ServiceMaster - registered UNIX signal handlers for [TERM, HUP, INT] 2018-04-20 16:44:15,149 [main] INFO service.ServiceMaster - HADOOP_CONF_DIR = /usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop 2018-04-20 16:44:15,149 [main] INFO service.ServiceMaster - JAVA_HOME = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre 2018-04-20 16:44:15,149 [main] INFO service.ServiceMaster - XFILESEARCHPATH = /usr/dt/app-defaults/%L/Dt 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - LANG = en_US.UTF-8 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - APP_SUBMIT_TIME_ENV = 1524242643815 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - NM_HOST = eyang-5.openstacklocal 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - HADOOP_HDFS_HOME = /usr/local/hadoop-3.2.0-SNAPSHOT 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - LOGNAME = hbase 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - JVM_PID = 23719 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - HADOOP_MAPRED_HOME = /usr/local/hadoop-3.2.0-SNAPSHOT 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - PWD = /tmp/hadoop-yarn/nm-local-dir/usercache/hbase/appcache/application_1524242413029_0001/container_1524242413029_0001_01_000001 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - HADOOP_COMMON_HOME = /usr/local/hadoop-3.2.0-SNAPSHOT 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - _ = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - LANGUAGE = en_US.UTF-8 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - LOCAL_DIRS = /tmp/hadoop-yarn/nm-local-dir/usercache/hbase/appcache/application_1524242413029_0001 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - APPLICATION_WEB_PROXY_BASE = /proxy/application_1524242413029_0001 2018-04-20 16:44:15,150 [main] INFO service.ServiceMaster - NM_HTTP_PORT = 8042 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - LOG_DIRS = /usr/local/hadoop-3.2.0-SNAPSHOT/logs/userlogs/application_1524242413029_0001/container_1524242413029_0001_01_000001 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - PRELAUNCH_OUT = /usr/local/hadoop-3.2.0-SNAPSHOT/logs/userlogs/application_1524242413029_0001/container_1524242413029_0001_01_000001/prelaunch.out 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - NM_AUX_SERVICE_mapreduce_shuffle = AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - NM_PORT = 54502 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - HADOOP_YARN_HOME = /usr/local/hadoop-3.2.0-SNAPSHOT 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - USER = hbase 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - CLASSPATH = yarnservice-log4j.properties:conf/:lib/*::/usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop 2018-04-20 16:44:15,151 [main] INFO service.ServiceMaster - PRELAUNCH_ERR = /usr/local/hadoop-3.2.0-SNAPSHOT/logs/userlogs/application_1524242413029_0001/container_1524242413029_0001_01_000001/prelaunch.err 2018-04-20 16:44:15,152 [main] INFO service.ServiceMaster - HADOOP_TOKEN_FILE_LOCATION = /tmp/hadoop-yarn/nm-local-dir/usercache/hbase/appcache/application_1524242413029_0001/container_1524242413029_0001_01_000001/container_tokens 2018-04-20 16:44:15,152 [main] INFO service.ServiceMaster - LC_ALL = en_US.UTF-8 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - NLSPATH = /usr/dt/lib/nls/msg/%L/%N.cat 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - LOCAL_USER_DIRS = /tmp/hadoop-yarn/nm-local-dir/usercache/hbase/ 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - HOME = /home/ 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - SHLVL = 1 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - CONTAINER_ID = container_1524242413029_0001_01_000001 2018-04-20 16:44:15,154 [main] INFO service.ServiceMaster - MALLOC_ARENA_MAX = 4 2018-04-20 16:44:16,362 [main] WARN shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2018-04-20 16:44:16,507 [main] INFO utils.ServiceApiUtil - Loading service definition from hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/abc.json 2018-04-20 16:44:16,966 [main] INFO service.ServiceMaster - class Service { name: abc id: application_1524242413029_0001 version: v1 description: null artifact: null resource: null launchTime: null numberOfRunningContainers: null lifetime: -1 components: [class Component { name: ping state: FLEXING dependencies: [] readinessCheck: null artifact: null launchCommand: sleep 1200000 resource: class Resource { profile: null cpus: 1 memory: 256 additional: {} } numberOfContainers: 2 containers: [] runPrivilegedContainer: false placementPolicy: null configuration: class Configuration { properties: {} env: {} files: [] } quicklinks: [] }] configuration: class Configuration { properties: {} env: {} files: [] } state: ACCEPTED quicklinks: {} queue: null kerberosPrincipal: class KerberosPrincipal { principalName: hbase/_HOST@EXAMPLE.COM keytab: file:///etc/security/keytabs/hbase.service.keytab } dockerClientConfig: null } 2018-04-20 16:44:17,093 [main] INFO service.ServiceMaster - Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1 cluster_timestamp: 1524242413029 } attemptId: 1 } keyId: -885639145) 2018-04-20 16:44:17,094 [main] INFO service.ServiceMaster - Kind: HDFS_DELEGATION_TOKEN, Service: 172.26.111.17:9000, Ident: (token for hbase: HDFS_DELEGATION_TOKEN owner=hbase, renewer=yarn, realUser=rm/eyang-1.openstacklocal@EXAMPLE.COM, issueDate=1524242643768, maxDate=1524847443768, sequenceNumber=1, masterKeyId=2) 2018-04-20 16:44:17,096 [main] INFO service.ServiceMaster - No keytab localized at keytabs/abc.keytab 2018-04-20 16:44:17,101 [main] INFO service.ServiceMaster - Using pre-installed keytab from localhost: file:///etc/security/keytabs/hbase.service.keytab 2018-04-20 16:44:17,101 [main] INFO service.ServiceMaster - User before logged in is: hbase (auth:SIMPLE) 2018-04-20 16:44:17,173 [main] INFO security.UserGroupInformation - Login successful for user hbase/eyang-5.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hbase.service.keytab 2018-04-20 16:44:17,173 [main] INFO service.ServiceMaster - User after logged in is: hbase/eyang-5.openstacklocal@EXAMPLE.COM (auth:KERBEROS) 2018-04-20 16:44:17,173 [main] INFO service.ServiceMaster - Remove HDFS delegation token Kind: HDFS_DELEGATION_TOKEN, Service: 172.26.111.17:9000, Ident: (token for hbase: HDFS_DELEGATION_TOKEN owner=hbase, renewer=yarn, realUser=rm/eyang-1.openstacklocal@EXAMPLE.COM, issueDate=1524242643768, maxDate=1524847443768, sequenceNumber=1, masterKeyId=2). 2018-04-20 16:44:17,202 [main] INFO service.ServiceMaster - Service AppAttemptId: appattempt_1524242413029_0001_000001 2018-04-20 16:44:17,327 [main] INFO beanutils.FluentPropertyBeanIntrospector - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. 2018-04-20 16:44:17,363 [main] INFO impl.MetricsConfig - loaded properties from hadoop-metrics2.properties 2018-04-20 16:44:17,431 [main] INFO impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s). 2018-04-20 16:44:17,431 [main] INFO impl.MetricsSystemImpl - ServiceAppMaster metrics system started 2018-04-20 16:44:17,442 [main] INFO service.ServiceScheduler - Set registry user accounts: sasl:hbase 2018-04-20 16:44:17,461 [main] INFO zk.RegistrySecurity - Registry default system acls: [1,s{'world,'anyone} , 31,s{'sasl,'yarn} , 31,s{'sasl,'jhs} , 31,s{'sasl,'hdfs-demo} , 31,s{'sasl,'rm} , 31,s{'sasl,'hive} ] 2018-04-20 16:44:17,462 [main] INFO zk.RegistrySecurity - Registry User ACLs [31,s{'sasl,'hbase} , 31,s{'sasl,'hbase} ] 2018-04-20 16:44:17,511 [main] INFO event.AsyncDispatcher - Registering class org.apache.hadoop.yarn.service.ServiceEventType for class org.apache.hadoop.yarn.service.ServiceScheduler$ServiceEventHandler 2018-04-20 16:44:17,511 [main] INFO event.AsyncDispatcher - Registering class org.apache.hadoop.yarn.service.component.ComponentEventType for class org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler 2018-04-20 16:44:17,512 [main] INFO event.AsyncDispatcher - Registering class org.apache.hadoop.yarn.service.component.instance.ComponentInstanceEventType for class org.apache.hadoop.yarn.service.ServiceScheduler$ComponentInstanceEventHandler 2018-04-20 16:44:17,541 [main] INFO impl.NMClientAsyncImpl - Upper bound of the thread pool size is 500 2018-04-20 16:44:17,544 [main] INFO service.ServiceMaster - Starting service as user hbase/eyang-5.openstacklocal@EXAMPLE.COM (auth:KERBEROS) 2018-04-20 16:44:17,561 [main] INFO ipc.CallQueueManager - Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2018-04-20 16:44:17,570 [Socket Reader #1 for port 43436] INFO ipc.Server - Starting Socket Reader #1 for port 43436 2018-04-20 16:44:17,608 [main] INFO pb.RpcServerFactoryPBImpl - Adding protocol org.apache.hadoop.yarn.service.impl.pb.service.ClientAMProtocolPB to the server 2018-04-20 16:44:17,632 [IPC Server Responder] INFO ipc.Server - IPC Server Responder: starting 2018-04-20 16:44:17,632 [main] INFO service.ClientAMService - Instantiated ClientAMService at eyang-5.openstacklocal/172.26.111.20:43436 2018-04-20 16:44:17,636 [IPC Server listener on 43436] INFO ipc.Server - IPC Server listener on 43436: starting 2018-04-20 16:44:17,636 [main] INFO zk.CuratorService - Creating CuratorService with connection fixed ZK quorum "eyang-1.openstacklocal:2181" 2018-04-20 16:44:17,648 [main] INFO zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = hbase/eyang-5.openstacklocal@EXAMPLE.COM, keytab = /etc/security/keytabs/hbase.service.keytab 2018-04-20 16:44:18,014 [main] INFO service.ServiceScheduler - Registering appattempt_1524242413029_0001_000001, abc into registry 2018-04-20 16:44:18,025 [main] INFO service.ServiceScheduler - Received 0 containers from previous attempt. 2018-04-20 16:44:18,073 [main] INFO service.ServiceScheduler - Found 2 containers from ZK registry: [ctr-1523993550287-0001-01-000003, ctr-1523993550287-0001-01-000007] 2018-04-20 16:44:18,073 [main] INFO registry.YarnRegistryViewForProviders - Resolving path /users/hbase/services/yarn-service/abc/components/ctr-1523993550287-0001-01-000003 2018-04-20 16:44:18,118 [main] INFO registry.YarnRegistryViewForProviders - Resolving path /users/hbase/services/yarn-service/abc/components/ctr-1523993550287-0001-01-000007 2018-04-20 16:44:18,130 [pool-5-thread-1] INFO service.ServiceScheduler - Registered service under /users/hbase/services/yarn-service/abc; absolute path /registry/users/hbase/services/yarn-service/abc 2018-04-20 16:44:18,133 [main] INFO service.ServiceScheduler - Triggering initial evaluation of component ping 2018-04-20 16:44:18,134 [main] INFO component.Component - [INIT COMPONENT ping]: 2 instances. 2018-04-20 16:44:18,134 [main] INFO component.Component - [COMPONENT ping] Requesting for 2 container(s) 2018-04-20 16:44:18,140 [main] INFO component.Component - [COMPONENT ping] Submitting container request : Capability[]Priority[0]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null] 2018-04-20 16:44:18,146 [main] INFO component.Component - [COMPONENT ping] Submitting container request : Capability[]Priority[0]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null] 2018-04-20 16:44:18,146 [main] INFO component.Component - [COMPONENT ping] Transitioned from INIT to FLEXING on FLEX event. 2018-04-20 16:44:20,107 [AMRM Callback Handler Thread] INFO service.ServiceScheduler - 2 containers allocated. 2018-04-20 16:44:20,107 [Component dispatcher] INFO component.Component - [COMPONENT ping]: container_1524242413029_0001_01_000002 allocated, num pending component instances reduced to 1 2018-04-20 16:44:20,108 [Component dispatcher] INFO component.Component - [COMPONENT ping]: Assigned container_1524242413029_0001_01_000002 to component instance ping-0 and launch on host eyang-4.openstacklocal:52497 2018-04-20 16:44:20,108 [AMRM Callback Handler Thread] INFO service.ServiceScheduler - [COMPONENT ping]: remove 2 outstanding container requests for allocateId 0 2018-04-20 16:44:20,108 [AMRM Callback Handler Thread] INFO service.ServiceScheduler - [COMPONENT ping]: remove 1 outstanding container requests for allocateId 0 2018-04-20 16:44:20,109 [Component dispatcher] INFO component.Component - [COMPONENT ping]: container_1524242413029_0001_01_000003 allocated, num pending component instances reduced to 0 2018-04-20 16:44:20,109 [Component dispatcher] INFO component.Component - [COMPONENT ping]: Assigned container_1524242413029_0001_01_000003 to component instance ping-1 and launch on host eyang-5.openstacklocal:54502 2018-04-20 16:44:20,131 [pool-6-thread-1] INFO provider.ProviderUtils - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002]: Creating dir on hdfs: hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/components/v1/ping/ping-0 2018-04-20 16:44:20,194 [pool-6-thread-1] INFO containerlaunch.ContainerLaunchService - launching container container_1524242413029_0001_01_000002 2018-04-20 16:44:20,195 [pool-6-thread-2] INFO provider.ProviderUtils - [COMPINSTANCE ping-1 : container_1524242413029_0001_01_000003]: Creating dir on hdfs: hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/components/v1/ping/ping-1 2018-04-20 16:44:20,202 [pool-6-thread-2] INFO containerlaunch.ContainerLaunchService - launching container container_1524242413029_0001_01_000003 2018-04-20 16:44:20,208 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] INFO impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for Container container_1524242413029_0001_01_000003 2018-04-20 16:44:20,208 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] INFO impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for Container container_1524242413029_0001_01_000002 2018-04-20 16:44:20,344 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-1 : container_1524242413029_0001_01_000003] Transitioned from INIT to STARTED on START event 2018-04-20 16:44:20,707 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] Transitioned from INIT to STARTED on START event 2018-04-20 16:44:22,382 [pool-5-thread-4] INFO instance.ComponentInstance - [COMPINSTANCE ping-1 : container_1524242413029_0001_01_000003] IP = [172.26.111.20], host = eyang-5.openstacklocal, cancel container status retriever 2018-04-20 16:44:23,764 [pool-5-thread-5] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] IP = [172.26.111.21], host = eyang-4.openstacklocal, cancel container status retriever 2018-04-20 16:44:48,157 [pool-7-thread-1] INFO monitor.ServiceMonitor - Readiness check succeeded for ping-0: Probe Status, time="Fri Apr 20 16:44:48 UTC 2018", outcome="success", message="Default probe: IP presence" 2018-04-20 16:44:48,158 [pool-7-thread-1] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] Transitioned from STARTED to READY on BECOME_READY event 2018-04-20 16:44:48,159 [pool-7-thread-1] INFO monitor.ServiceMonitor - Readiness check succeeded for ping-1: Probe Status, time="Fri Apr 20 16:44:48 UTC 2018", outcome="success", message="Default probe: IP presence" 2018-04-20 16:44:48,159 [pool-7-thread-1] INFO component.Component - [COMPONENT ping] state changed from FLEXING -> STABLE 2018-04-20 16:44:48,159 [pool-7-thread-1] INFO service.ServiceMaster - Service state changed from STARTED -> STABLE 2018-04-20 16:44:48,159 [pool-7-thread-1] INFO instance.ComponentInstance - [COMPINSTANCE ping-1 : container_1524242413029_0001_01_000003] Transitioned from STARTED to READY on BECOME_READY event 2018-04-20 16:44:48,160 [Component dispatcher] INFO component.Component - [COMPONENT ping] Transitioned from FLEXING to STABLE on CHECK_STABLE event. 2018-04-20 16:45:25,325 [Socket Reader #1 for port 43436] INFO ipc.Server - Auth successful for rm/eyang-1.openstacklocal@EXAMPLE.COM (auth:KERBEROS) 2018-04-20 16:45:25,357 [Socket Reader #1 for port 43436] INFO authorize.ServiceAuthorizationManager - Authorization successful for hbase (auth:PROXY) via rm/eyang-1.openstacklocal@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.service.ClientAMProtocol 2018-04-20 16:45:25,541 [IPC Server handler 0 on 43436] INFO service.ClientAMService - Upgrading service to version v2 by hbase (auth:PROXY) via rm/eyang-1.openstacklocal@EXAMPLE.COM (auth:KERBEROS) 2018-04-20 16:45:25,542 [Component dispatcher] INFO utils.ServiceApiUtil - Loading service definition from hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/upgrade/v2/abc.json 2018-04-20 16:45:25,570 [Component dispatcher] INFO service.ServiceManager - [SERVICE] Transitioned from STABLE to UPGRADING on UPGRADE event. 2018-04-20 16:45:25,571 [Component dispatcher] INFO component.Component - [COMPONENT ping] Transitioned from STABLE to UPGRADING on UPGRADE event. 2018-04-20 16:50:14,245 [Socket Reader #1 for port 43436] INFO ipc.Server - Auth successful for rm/eyang-1.openstacklocal@EXAMPLE.COM (auth:KERBEROS) 2018-04-20 16:50:14,249 [Socket Reader #1 for port 43436] INFO authorize.ServiceAuthorizationManager - Authorization successful for hbase (auth:PROXY) via rm/eyang-1.openstacklocal@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.service.ClientAMProtocol 2018-04-20 16:50:14,288 [IPC Server handler 0 on 43436] INFO service.ClientAMService - Upgrade container container_1524242413029_0001_01_000002 2018-04-20 16:50:14,290 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] Transitioned from READY to UPGRADING on UPGRADE event 2018-04-20 16:50:14,309 [pool-6-thread-3] INFO provider.ProviderUtils - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002]: Creating dir on hdfs: hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/components/v2/ping/ping-0 2018-04-20 16:50:14,327 [pool-6-thread-3] INFO containerlaunch.ContainerLaunchService - reInitializing container container_1524242413029_0001_01_000002 2018-04-20 16:50:14,331 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2] INFO impl.NMClientAsyncImpl - Processing Event EventType: REINITIALIZE_CONTAINER for Container container_1524242413029_0001_01_000002 2018-04-20 16:50:14,379 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] Transitioned from UPGRADING to READY on BECOME_READY event 2018-04-20 16:50:16,726 [AMRM Callback Handler Thread] WARN service.ServiceScheduler - Nodes updated info: eyang-4.openstacklocal:52497, state = UNHEALTHY, healthDiagnostics = Linux Container Executor reached unrecoverable exception 2018-04-20 16:50:16,730 [Component dispatcher] INFO component.Component - [COMPONENT ping] Requesting for 1 container(s) 2018-04-20 16:50:16,730 [Component dispatcher] INFO component.Component - [COMPONENT ping] Submitting container request : Capability[]Priority[0]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null] 2018-04-20 16:50:16,734 [pool-5-thread-1] INFO registry.YarnRegistryViewForProviders - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002]: Deleting registry path /users/hbase/services/yarn-service/abc/components/ctr-1524242413029-0001-01-000002 2018-04-20 16:50:16,736 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002]: container_1524242413029_0001_01_000002 completed. Reinsert back to pending list and requested a new container. exitStatus=-100, diagnostics=Container released on a *lost* node. 2018-04-20 16:50:16,736 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE ping-0 : container_1524242413029_0001_01_000002] Transitioned from READY to INIT on STOP event 2018-04-20 16:50:16,739 [Component dispatcher] ERROR component.Component - [COMPONENT ping]: Invalid event CHECK_STABLE at UPGRADING org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: CHECK_STABLE at UPGRADING at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487) at org.apache.hadoop.yarn.service.component.Component.handle(Component.java:839) at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:573) at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:562) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:748) 2018-04-20 16:50:18,746 [AMRM Callback Handler Thread] INFO service.ServiceScheduler - 1 containers allocated. 2018-04-20 16:50:18,746 [AMRM Callback Handler Thread] INFO service.ServiceScheduler - [COMPONENT ping]: remove 1 outstanding container requests for allocateId 0 2018-04-20 16:50:18,747 [Component dispatcher] ERROR component.Component - [COMPONENT ping]: Invalid event CONTAINER_ALLOCATED at UPGRADING org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: CONTAINER_ALLOCATED at UPGRADING at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487) at org.apache.hadoop.yarn.service.component.Component.handle(Component.java:839) at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:573) at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:562) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:748) 2018-04-20 17:03:33,376 [AMRM Callback Handler Thread] WARN service.ServiceScheduler - Container container_1524242413029_0001_01_000004 Completed. No component instance exists. exitStatus=-100. diagnostics=Container expired since it was unused 2018-04-20 17:34:14,021 [AMRM Heartbeater thread] INFO impl.AMRMClientAsyncImpl - Shutdown requested. Stopping callback. 2018-04-20 17:34:15,834 [SIGTERM handler] ERROR service.ServiceMaster - RECEIVED SIGNAL 15: SIGTERM 2018-04-20 17:34:15,837 [pool-1-thread-1] INFO service.ServiceMaster - Stopping app master 2018-04-20 17:34:15,838 [pool-1-thread-1] INFO service.ServiceScheduler - Stopping service scheduler 2018-04-20 17:34:15,840 [pool-1-thread-1] INFO impl.MetricsSystemImpl - Stopping ServiceAppMaster metrics system... 2018-04-20 17:34:15,842 [pool-1-thread-1] INFO impl.MetricsSystemImpl - ServiceAppMaster metrics system stopped. 2018-04-20 17:34:15,842 [pool-1-thread-1] INFO impl.MetricsSystemImpl - ServiceAppMaster metrics system shutdown complete. 2018-04-20 17:34:15,842 [pool-1-thread-1] INFO event.AsyncDispatcher - AsyncDispatcher is draining to stop, ignoring any new events. 2018-04-20 17:34:15,869 [pool-1-thread-1] INFO ipc.Server - Stopping server on 43436 2018-04-20 17:34:15,870 [IPC Server listener on 43436] INFO ipc.Server - Stopping IPC Server listener on 43436 2018-04-20 17:34:15,870 [IPC Server Responder] INFO ipc.Server - Stopping IPC Server Responder 2018-04-20 17:34:15,871 [pool-1-thread-1] INFO service.ServiceMaster - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ServiceMaster at eyang-5.openstacklocal/172.26.111.20 ************************************************************/