Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.5.0
-
None
-
None
-
None
Description
Commit fails then Tez tries to recover which fails again.
5499174247-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Found summary file in attempt directory, summaryFile=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1 5499174696-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Using hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1 for recovering data from previous attempt 5499174963-2015-04-01 16:23:20,690 INFO [main] app.RecoveryParser: Parsing summary file, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, len=4024, lastModTime=1427919788998 5499175254-2015-04-01 16:23:20,786 INFO [main] app.RecoveryParser: Reached end of summary stream 5499175340-2015-04-01 16:23:21,087 INFO [main] app.RecoveryParser: Checking if DAG is in recoverable state, dagId=dag_1426707664723_1086_1 5499175468-2015-04-01 16:23:21,088 WARN [main] app.RecoveryParser: Found last inProgress DAG but not recoverable: dagId=dag_1426707664723_1086_1, dagCompleted=false 5499175622-2015-04-01 16:23:21,088 INFO [main] app.RecoveryParser: Trying to recover dag from recovery file, dagId=dag_1426707664723_1086_1, dataDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1, intoCurrentDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2 5499176102-2015-04-01 16:23:21,091 INFO [main] app.RecoveryParser: Copying DAG data into Current Attempt directory, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dag_1426707664723_1086_1.recovery 5499176413-2015-04-01 16:23:21,211 INFO [main] app.RecoveryParser: Recovering from event, eventType=DAG_SUBMITTED, event=dagID=dag_1426707664723_1086_1, submitTime=1427917169723 5499176580-2015-04-01 16:23:21,309 INFO [main] app.DAGAppMaster: Generating DAG graphviz file, dagId=dag_1426707664723_1086_1, filePath=/grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1.dot 5499176829-2015-04-01 16:23:21,347 INFO [main] app.DAGAppMaster: Writing DAG plan to: /grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1-tez-dag.pb.txt 5499177039-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Finished copying data from previous attempt into current attempt 5499177160-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Trying to create data recovered flag file, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dataRecovered 5499177445-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: In Session mode. Waiting for DAG over RPC 5499177541-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: Found previous DAG in completed or non-recoverable state, dagId=dag_1426707664723_1086_1, isCompleted=false, isNonRecoverable=true, state=null, failureReason=DAG Commit was in progress, not recoverable, dagId=dag_1426707664723_1086_1 5499177829-2015-04-01 16:23:22,601 INFO [main] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1 5499177953- 5499177954-LogType:syslog_dag_1426707664723_1086_1 5499177994-Log Upload Time:1-Apr-2015 16:24:30 5499178030-LogLength:521 5499178044-Log Contents: 5499178058-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: Recovered DAG: dag_1426707664723_1086_1 finished with state: FAILED 5499178176-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: dag_1426707664723_1086_1 transitioned from NEW to FAILED 5499178283-2015-04-01 16:23:22,604 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1426707664723_1086_1, dagState=FAILED 5499178425-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1_post 5499178579- 5499178580-LogType:syslog_dag_1426707664723_1086_1_post 5499178625-Log Upload Time:1-Apr-2015 16:24:30 5499178661-LogLength:4021 5499178676-Log Contents: 5499178690-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] app.DAGAppMaster: Waiting for next DAG to be submitted. 5499178807-2015-04-01 16:24:01,681 INFO [IPC Server handler 0 on 53890] client.DAGClientHandler: Received message to shutdown AM 5499178925-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] rm.TaskSchedulerEventHandler: TaskScheduler notified that it should unregister from RM 5499179073-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: No current running DAG, shutting down the AM 5499179197-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: DAGAppMasterShutdownHandler invoked 5499179312-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: Handling DAGAppMaster shutdown 5499179422-2015-04-01 16:24:01,683 INFO [AMShutdownThread] app.DAGAppMaster: Sleeping for 5 seconds before shutting down 5499179532-2015-04-01 16:24:04,151 INFO [HistoryEventHandlingThread] ats.ATSHistoryLoggingService: Event queue stats, eventsProcessedSinceLastUpdate=2, eventQueueSize=0 5499179690-2015-04-01 16:24:06,683 INFO [AMShutdownThread] app.DAGAppMaster: Calling stop for all the services 5499179790-2015-04-01 16:24:06,686 INFO [AMShutdownThread] history.HistoryEventHandler: Stopping HistoryEventHandler 5499179896-2015-04-01 16:24:06,686 INFO [AMShutdownThread] recovery.RecoveryService: Stopping RecoveryService 5499179995-2015-04-01 16:24:06,686 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0 5499180114-2015-04-01 16:24:06,686 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning 5499180238-2015-04-01 16:24:06,692 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted 5499180367-2015-04-01 16:24:06,697 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=0, failedDAGs=1, killedDAGs=0 5499180589-, trackingURL= 5499180604-2015-04-01 16:24:06,713 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 5499180730-2015-04-01 16:24:06,819 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM 5499180853-2015-04-01 16:24:06,821 INFO [AMShutdownThread] ipc.Server: Stopping server on 50998 5499180938-2015-04-01 16:24:06,821 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue 5499181060:java.lang.InterruptedException 5499181091- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) 5499181228- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) 5499181346- at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 5499181426- at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) 5499181551-2015-04-01 16:24:06,823 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 5499181645-2015-04-01 16:24:06,822 INFO [IPC Server listener on 50998] ipc.Server: Stopping IPC Server listener on 50998 5499181755-2015-04-01 16:24:06,822 INFO [AMShutdownThread] ipc.Server: Stopping server on 53890 5499181840-2015-04-01 16:24:06,826 INFO [IPC Server listener on 53890] ipc.Server: Stopping IPC Server listener on 53890 5499181950-2015-04-01 16:24:06,826 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 5499182044-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked 5499182135-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: The shutdown handler is still running, waiting for it to complete 5499182259-2015-04-01 16:24:06,839 WARN [AMShutdownThread] app.DAGAppMaster: Failed to delete tez scratch data dir, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086 5499182521-2015-04-01 16:24:06,839 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye! 5499182618-2015-04-01 16:24:06,839 INFO [Thread-1] app.DAGAppMaster: The shutdown handler has completed 5499182711- 5499182712- 5499182713- 5499182714-Container: container_1426707664723_1086_01_000304 on cn110-10.l42scl.hortonworks.com_45454 5499182805-============================================================================================ 5499182898-LogType:stderr 5499182913-Log Upload Time:1-Apr-2015 16:24:30 5499182949-LogLength:0 5499182961-Log Contents: 5499182975- 5499182976-LogType:stdout 5499182991-Log Upload Time:1-Apr-2015 16:24:30 5499183027-LogLength:81042 5499183043-Log Contents: 5499183057-9.076: [GC [PSYoungGen: 415194K->108031K(756736K)] 1463770K->1182454K(7674880K), 0.1355580 secs] [Times: user=0.59 sys=0.13, real=0.13 secs] 5499183199-16.885: [GC [PSYoungGen: 446979K->70470K(756736K)] 1521402K->1144901K(7674880K), 0.1141880 secs] [Times: user=0.41 sys=0.08, real=0.11 secs] 5499183341-32.610: [GC [PSYoungGen: 401811K->23054K(756736K)] 1476242K->1097493K(7674880K), 0.0198940 secs] [Times: user=0.26 sys=0.00, real=0.02 secs] 5499183483-37.397: [GC [PSYoungGen: 354557K->108009K(756736K)] 2477572K->2246623K(7674880K), 0.0591560 secs] [Times: user=0.82 sys=0.06, real=0.06 secs] 5499183626-42.607: [GC [PSYoungGen: 434535K->15928K(649216K)] 2573148K->2154549K(7567360K), 0.0639210 secs] [Times: user=0.45 sys=0.03, real=0.06 secs] 5499183768-47.641: [GC [PSYoungGen: 553804K->22893K(689152K)] 3741001K->3210098K(7607296K), 0.0727270 secs] [Times: user=0.55 sys=0.06, real=0.07 secs] 5499183910-53.107: [GC [PSYoungGen: 343874K->137457K(646656K)] 4579655K->4384105K(7564800K), 0.1012020 secs] [Times: user=0.50 sys=0.14, real=0.10 secs] 5499184053-62.586: [GC [PSYoungGen: 484074K->89878K(676864K)] 4730722K->4336778K(7595008K), 0.0985300 secs] [Times: user=0.56 sys=0.08, real=0.10 secs] 5499184195-76.005: [GC [PSYoungGen: 449294K->16827K(668672K)] 4696194K->4274117K(7586816K), 0.0315610 secs] [Times: user=0.36 sys=0.03, real=0.03 secs] 5499184337-79.777: [GC [PSYoungGen: 211179K->19599K(680448K)] 5517045K->5332144K(7598592K), 0.0304100 secs] [Times: user=0.37 sys=0.02, real=0.03 secs] 5499184479-81.315: [GC [PSYoungGen: 150008K->78090K(677888K)] 5462554K->5399597K(7596032K), 0.0293570 secs] [Times: user=0.39 sys=0.02, real=0.03 secs] 5499184621-82.455: [GC [PSYoungGen: 237557K->512K(687616K)] 5559064K->5324779K(7605760K), 0.0384990 secs] [Times: user=0.34 sys=0.02, real=0.04 secs] 5499184761-84.067: [GC [PSYoungGen: 210827K->14117K(687616K)] 6583671K->6387049K(7605760K), 0.0517180 secs] [Times: user=0.66 sys=0.01, real=0.05 secs] 5499184903-88.416: [GC [PSYoungGen: 268920K->15351K(696320K)] 6641851K->6394989K(7614464K), 0.0787950 secs] [Times: user=0.51 sys=0.02, real=0.08 secs] 5499185045-101.043: [GC [PSYoungGen: 376721K->448K(691200K)] 6756359K->6387846K(7609344K), 0.0282280 secs] [Times: user=0.44 sys=0.02, real=0.03 secs] 5499185186-103.105: [GC [PSYoungGen: 82965K->13643K(705536K)] 6470363K->6401129K(7623680K), 0.1482160 secs] [Times: user=0.53 sys=0.00, real=0.15 secs] 5499185328-103.253: [GC [PSYoungGen: 13643K->0K(699392K)] 6401129K->6400834K(7617536K), 0.0805200 secs] [Times: user=0.56 sys=0.02, real=0.08 secs] 5499185466-103.334: [Full GC [PSYoungGen: 0K->0K(699392K)] [ParOldGen: 6400834K->26007K(6918144K)] 6400834K->26007K(7617536K) [PSPermGen: 33081K->33057K(66560K)], 0.4079400 secs] [Times: user=1.08 sys=0.17, real=0.41 secs] 5499185679-108.478: [GC [PSYoungGen: 279515K->23409K(718848K)] 1354098K->1097992K(7636992K), 0.0143960 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 5499185822-121.280: [GC [PSYoungGen: 322641K->279K(709632K)] 1397224K->1082530K(7627776K), 0.0187460 secs] [Times: user=0.10 sys=0.00, real=0.01 secs] 5499185963-126.595: [GC [PSYoungGen: 345663K->20073K(731648K)] 2476491K->2150964K(7649792K), 0.0305890 secs] [Times: user=0.19 sys=0.00, real=0.03 secs] 5499186106-141.120: [GC [PSYoungGen: 596191K->14241K(723456K)] 3775658K->3200292K(7641600K), 0.0189990 secs] [Times: user=0.27 sys=0.00, real=0.02 secs] -- 18043554745-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 49 18043554860-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 44 18043554975-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 45 18043555090-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 50 18043555205-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 25 18043555320-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 17 18043555435-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 23 18043555550-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 24 18043555665-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 21 18043555780-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 22 18043555895-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 38 18043556014-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 20 18043556129-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 31 18043556248-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 37 18043556367-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 29 18043556482-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 30 18043556601-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 27 18043556716-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 28 18043556831-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 3 18043556949-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 26 18043557064-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 2 18043557182-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 32 18043557297-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 36 18043557416-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 35 18043557535-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 34 18043557654-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 33 18043557769-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 1 18043557883-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 39 18043557998-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 5 18043558112-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 4 18043558226-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 43 18043558341-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 53 18043558456-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 42 18043558571-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 54 18043558686-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 41 18043558801-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 51 18043558916-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 14 18043559031-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 7 18043559149-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 13 18043559264-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 6 18043559382-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 16 18043559497-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 19 18043559612-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 18 18043559727-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 15 18043559842-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No exclusive output committers for vertex: Reducer 12 18043559971-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 10 18043560090-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 11 18043560209-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 8 18043560327-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 9 18043560445-2015-04-01 16:23:08,857 FATAL [AsyncDispatcher event handler] event.AsyncDispatcher: Error in dispatcher thread 18043560557:org.apache.tez.common.counters.LimitExceededException: Too many counters: 1201 max=1200 18043560645- at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87) 18043560717- at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94) 18043560788- at org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:75) 18043560885- at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92) 18043560986- at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:103) 18043561085- at org.apache.tez.common.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:198) 18043561188- at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:363) 18043561283- at org.apache.tez.dag.app.dag.impl.DAGImpl.incrTaskCounters(DAGImpl.java:598) 18043561362- at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:588) 18043561439- at org.apache.tez.dag.app.dag.impl.DAGImpl.logJobHistoryFinishedEvent(DAGImpl.java:994) 18043561528- at org.apache.tez.dag.app.dag.impl.DAGImpl.finished(DAGImpl.java:1135) 18043561600- at org.apache.tez.dag.app.dag.impl.DAGImpl.checkDAGForCompletion(DAGImpl.java:1048) 18043561685- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1708) 18043561785- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1665) 18043561885- at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) 18043562001- at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) 18043562097- at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) 18043562190- at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) 18043562307- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:944) 18043562376- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:126) 18043562445- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1686) 18043562535- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1677) 18043562625- at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) 18043562709- at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) 18043562790- at java.lang.Thread.run(Thread.java:745) 18043562832-2015-04-01 16:23:08,882 INFO [AsyncDispatcher event handler] event.AsyncDispatcher: Exiting, bbye.. 18043562932-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked 18043563023-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMaster received a signal. Signaling TaskScheduler 18043563137-2015-04-01 16:23:08,885 INFO [Thread-1] rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was : true 18043563257-2015-04-01 16:23:08,899 INFO [Thread-1] history.HistoryEventHandler: Stopping HistoryEventHandler 18043563355-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Stopping RecoveryService 18043563446-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Closing Summary Stream 18043563535-2015-04-01 16:23:08,900 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning 18043563659-2015-04-01 16:23:09,033 INFO [Thread-1] recovery.RecoveryService: Closing Output Stream for DAG dag_1426707664723_1086_1 18043563780-2015-04-01 16:23:09,062 INFO [Thread-1] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0 18043563891-2015-04-01 16:23:09,064 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000319 18043564052-2015-04-01 16:23:09,064 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454 18043564185-2015-04-01 16:23:09,097 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000047 18043564346-2015-04-01 16:23:09,097 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn122-10.l42scl.hortonworks.com:45454 18043564479-2015-04-01 16:23:09,114 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000306 18043564640-2015-04-01 16:23:09,114 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454 18043564773-2015-04-01 16:23:09,120 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000104 18043564934-2015-04-01 16:23:09,120 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn111-10.l42scl.hortonworks.com:45454 18043565067-2015-04-01 16:23:09,145 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000140 18043565228-2015-04-01 16:23:09,145 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn120-10.l42scl.hortonworks.com:45454 18043565361-2015-04-01 16:23:09,152 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000236 18043565522-2015-04-01 16:23:09,152 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn107-10.l42scl.hortonworks.com:45454 18043565655-2015-04-01 16:23:09,159 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000255 18043565816-2015-04-01 16:23:09,159 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn116-10.l42scl.hortonworks.com:45454 18043565949-2015-04-01 16:23:09,182 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000074