Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9612

DistCp worker threads are not terminated after jobs are done.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HADOOP-11827, a producer-consumer style thread pool was introduced to parallelize the task of listing files/directories.

      We have a use case where a distcp job is run during the commit phase of a MR2 job. However, it was found distcp does not terminate ProducerConsumer thread pools properly. Because threads are not terminated, those MR2 jobs never finish.

      In a more typical use case where distcp is run as a standalone job, those threads are terminated forcefully when the java process is terminated. So these leaked threads did not become a problem.

        Attachments

        1. HDFS-9612.009.patch
          8 kB
          Yongjun Zhang
        2. HDFS-9612.008.patch
          8 kB
          Wei-Chiu Chuang
        3. HDFS-9612.007.patch
          8 kB
          Wei-Chiu Chuang
        4. HDFS-9612.006.patch
          7 kB
          Wei-Chiu Chuang
        5. HDFS-9612.005.patch
          7 kB
          Wei-Chiu Chuang
        6. HDFS-9612.004.patch
          7 kB
          Wei-Chiu Chuang
        7. HDFS-9612.003.patch
          7 kB
          Wei-Chiu Chuang
        8. HDFS-9612.002.patch
          5 kB
          Wei-Chiu Chuang
        9. HDFS-9612.001.patch
          17 kB
          Wei-Chiu Chuang

          Issue Links

            Activity

              People

              • Assignee:
                jojochuang Wei-Chiu Chuang
                Reporter:
                jojochuang Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: