[SAMZA-2198] Container process always takes task.shutdown.ms to shut down

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0, 1.1
• Fix Version/s: 1.3
• Assignee: Abhishek Shivanna
• Reporter: Cameron Lee
• Created: 17/May/19 21:41
• Updated: 07/Nov/19 00:05
• Resolved: 19/Sep/19 00:29

Description

https://issues.apache.org/jira/browse/SAMZA-1724 added a System.exit to LocalContainerRunner (now moved to ContainerLaunchUtil) in order to ensure the process exits. However, this seems to cause a partial deadlock when shutting down the container process. SamzaContainer adds a shutdown hook which joins on the run loop thread (which is also the main thread). During the shutdown process, when the main thread reaches the System.exit in ContainerLaunchUtil, it looks like it waits for the shutdown hook to complete. However, the shutdown hook is waiting for the main thread to join. Therefore, the shutdown hook will always wait task.shutdown.ms before giving up on joining the main thread, and then the process exits.

This issue was not observed in a job on 0.14. The System.exit was only added in 1.0.

Issue Links

• links to: GitHub Pull Request #1151