2015-09-07 15:31:41,093 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1441194300243_337619_000001 (auth:SIMPLE) 2015-09-07 15:31:41,094 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1441194300243_337619_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB 2015-09-07 15:31:41,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1441194300243_337619_01_000001 by user dd_edw 2015-09-07 15:31:41,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1441194300243_337619 2015-09-07 15:31:41,094 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=dd_edw IP=172.22.168.103 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1441194300243_337619 CONTAINERID=container_1441194300243_337619_01_000001 2015-09-07 15:31:41,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1441194300243_337619 transitioned from NEW to INITING 2015-09-07 15:31:41,095 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1441194300243_337619_01_000001 to application application_1441194300243_337619 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1441194300243_337619 transitioned from INITING to RUNNING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1441194300243_337619_01_000001 transitioned from NEW to LOCALIZING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1441194300243_337619 2015-09-07 15:31:41,171 INFO org.apache.spark.network.yarn.YarnShuffleService: Initializing container container_1441194300243_337619_01_000001 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GetUVVisitor.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFFullToHalf.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/udffrom173.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFPageRegexpN.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetRand.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GeometricMean.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGeoComUtils.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFNewBand.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFIPToArea.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/RowNumberUDF.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGeoUtilsNew.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GetMD5UDF.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFTimeToEST.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetCommision.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFNumberToString.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/collect_all.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDAFGroupConn.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetIndex.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFStickFunction.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFLookUpMapping.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/EncryptIndexUDF.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/SysDate.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFUnionGetTime.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetUserAgent.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/IndexChange.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UserAgentUtils-1.8.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetEndTime.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,172 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/HexadecimalGB.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1:8020/user/dd_edw/.staging/job_1441194300243_337619/job.xml transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1:8020/user/dd_edw/.staging/job_1441194300243_337619/job.jar transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1:8020/user/dd_edw/.staging/job_1441194300243_337619/job.splitmetainfo transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1:8020/user/dd_edw/.staging/job_1441194300243_337619/job.split transitioned from INIT to DOWNLOADING 2015-09-07 15:31:41,173 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1441194300243_337619_01_000001 2015-09-07 15:31:41,174 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /data6/yarn1/local/nmPrivate/container_1441194300243_337619_01_000001.tokens. Credentials list: 2015-09-07 15:37:47,492 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /data6/yarn1/local/nmPrivate/container_1441194300243_337619_01_000001.tokens to /data0/yarn1/local/usercache/dd_edw/appcache/application_1441194300243_337619/container_1441194300243_337619_01_000001.tokens 2015-09-07 15:37:47,492 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /data0/yarn1/local/usercache/dd_edw/appcache/application_1441194300243_337619 = file:/data0/yarn1/local/usercache/dd_edw/appcache/application_1441194300243_337619 2015-09-07 15:37:47,605 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GetUVVisitor.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,618 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFFullToHalf.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/udffrom173.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,645 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFPageRegexpN.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,657 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetRand.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GeometricMean.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,682 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGeoComUtils.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,701 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFNewBand.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFIPToArea.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:37:47,738 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/RowNumberUDF.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:40:48,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGeoUtilsNew.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:40:48,590 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/GetMD5UDF.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:40:48,602 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFTimeToEST.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:40:48,739 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://ns1/user/dd_edw/.staging/job_1441194300243_337619/libjars/UDFGetCommision.jar transitioned from DOWNLOADING to LOCALIZED 2015-09-07 15:41:43,650 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1441194300243_337619_000001 (auth:SIMPLE) 2015-09-07 15:41:43,661 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1441194300243_337619_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB 2015-09-07 15:41:43,662 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1441194300243_337619_01_000001 2015-09-07 15:41:43,662 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=dd_edw IP=172.22.168.103 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1441194300243_337619 CONTAINERID=container_1441194300243_337619_01_000001 2015-09-07 15:41:43,662 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1441194300243_337619_01_000001 transitioned from LOCALIZING to KILLING 2015-09-07 15:41:43,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1441194300243_337619_01_000001 transitioned from KILLING to DONE 2015-09-07 15:41:43,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1441194300243_337619_01_000001 from application application_1441194300243_337619 2015-09-07 15:41:43,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1441194300243_337619_01_000001 for log-aggregation 2015-09-07 15:41:43,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1441194300243_337619 2015-09-07 15:41:43,663 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed container container_1441194300243_337619_01_000001 2015-09-07 15:41:43,663 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping container container_1441194300243_337619_01_000001 2015-09-07 15:41:43,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1441194300243_337619 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2015-09-07 15:41:43,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1441194300243_337619 2015-09-07 15:41:43,668 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping application application_1441194300243_337619 2015-09-07 15:41:43,668 INFO org.apache.spark.network.shuffle.ExternalShuffleBlockManager: Application application_1441194300243_337619 removed, cleanupLocalDirs = false 2015-09-07 15:41:43,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1441194300243_337619 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2015-09-07 15:41:43,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1441194300243_337619 2015-09-07 15:41:43,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Starting aggregate log-file for app application_1441194300243_337619 at /tmp/app-logs/dd_edw/logs/application_1441194300243_337619/BJHC-HERA-16517.hadoop.jd.local_50086.tmp 2015-09-07 15:41:43,733 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1441194300243_337619_01_000001. Current good log dirs are /data0/yarn1/logs,/data1/yarn1/logs,/data2/yarn1/logs,/data3/yarn1/logs,/data4/yarn1/logs,/data5/yarn1/logs,/data6/yarn1/logs,/data7/yarn1/logs,/data8/yarn1/logs,/data9/yarn1/logs,/data10/yarn1/logs,/data11/yarn1/logs 2015-09-07 15:41:43,793 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Finished aggregate log-file for app application_1441194300243_337619 2015-09-07 15:41:44,176 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1441194300243_337619_01_000001