[HDFS-9612] DistCp worker threads are not terminated after jobs are done.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: distcp
• Hadoop Flags: Reviewed
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 04/Jan/16 22:36
• Updated: 02/Oct/19 17:15
• Resolved: 15/Jan/16 19:02

Description

In HADOOP-11827, a producer-consumer style thread pool was introduced to parallelize the task of listing files/directories.

We have a use case where a distcp job is run during the commit phase of a MR2 job. However, it was found distcp does not terminate ProducerConsumer thread pools properly. Because threads are not terminated, those MR2 jobs never finish.

In a more typical use case where distcp is run as a standalone job, those threads are terminated forcefully when the java process is terminated. So these leaked threads did not become a problem.

Issue Links

• is broken by: HADOOP-11827 Speed-up distcp buildListing() using threadpool
• links to: Java 7 Executor Service explained