[TEZ-1629] Replace ThreadPool's default RejectedExecutionHandler in ContainerLauncherImpl to void abort when AM shutdown

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.2
• Assignee: Jeff Zhang
• Reporter: Rajesh Balamohan
• Created: 29/Sep/14 13:05
• Updated: 08/Nov/14 20:25
• Resolved: 22/Oct/14 05:25

Description

2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.DAGAppMaster: On DAG completion. Old state: RUNNING new state: SUCCEEDED
2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.DAGAppMaster: Shutting down on completion of dag:dag_1410976766734_0795_1
2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.DAGAppMaster: DAGAppMasterShutdownHandler invoked
2014-09-19 10:18:42,655 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.DAGAppMaster: Handling DAGAppMaster shutdown
2014-09-19 10:18:42,656 INFO [AMShutdownThread] org.apache.tez.dag.app.DAGAppMaster: Sleeping for 5 seconds before shutting down
2014-09-19 10:18:47,661 INFO [AMShutdownThread] org.apache.tez.dag.app.DAGAppMaster: Calling stop for all the services
2014-09-19 10:18:47,662 INFO [AMShutdownThread] org.apache.tez.dag.history.HistoryEventHandler: Stopping HistoryEventHandler
2014-09-19 10:18:47,663 INFO [AMShutdownThread] org.apache.tez.dag.history.recovery.RecoveryService: Stopping RecoveryService
2014-09-19 10:18:47,663 INFO [AMShutdownThread] org.apache.tez.dag.history.recovery.RecoveryService: Closing Summary Stream
2014-09-19 10:18:47,663 INFO [RecoveryEventHandlingThread] org.apache.tez.dag.history.recovery.RecoveryService: EventQueue take interrupted. Returning
2014-09-19 10:18:47,699 INFO [AMShutdownThread] org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
2014-09-19 10:18:47,699 INFO [HistoryEventHandlingThread] org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService: EventQueue take interrupted. Returning
2014-09-19 10:18:47,700 INFO [AMShutdownThread] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1410976766734_0795_01_000002
2014-09-19 10:18:47,711 INFO [AsyncDispatcher event handler] org.apache.tez.dag.history.HistoryEventHandler: [HISTORY][DAG:dag_1410976766734_0795_1][Event:CONTAINER_STOPPED]: containerId=container_1410976766734_0795_01_000002, stoppedTime=1411121927711, exitStatus=-100
2014-09-19 10:18:47,712 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.rm.container.AMContainerImpl: AMContainer container_1410976766734_0795_01_000002 transitioned from IDLE to STOP_REQUESTED via event C_NM_STOP_SENT
2014-09-19 10:18:47,713 INFO [DelayedContainerManager] org.apache.tez.dag.app.rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
2014-09-19 10:18:47,713 ERROR [ContainerLauncher Event Handler] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[ContainerLauncher Event Handler,5,main] threw an Exception.
java.util.concurrent.RejectedExecutionException: Task org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor@8b1f182 rejected from java.util.concurrent.ThreadPoolExecutor@5a669243[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$1.run(ContainerLauncherImpl.java:310)