Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14295

Add Threadpool for DataTransfers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      When a DataNode data transfers a block, is spins up a new thread for each transfer.  Here and Here.  Instead, add the threads to a CachedThreadPool so that when their threads complete the transfer, they can be re-used for another transfer. This should save resources spent on creating and spinning up transfer threads.

      One thing I'll point out that's a bit off, which I address in this patch, ...

      There are two places in the code where a DataTransfer thread is started. In one place, it's started in a default thread group. In another place, it's started in the dataXceiverServer thread group.

      I do not think it's correct to include any of these threads in the dataXceiverServer thread group. Anything submitted to the dataXceiverServer should probably be tied to the dfs.datanode.max.transfer.threads configurations, and neither of these methods are. Instead, they should be submitted into the same thread pool with its own thread group (probably the default thread group, unless someone suggests otherwise) and is what I have included in this patch.

        Attachments

        1. HDFS-14295.1.patch
          4 kB
          David Mollitor
        2. HDFS-14295.2.patch
          5 kB
          David Mollitor
        3. HDFS-14295.3.patch
          5 kB
          David Mollitor
        4. HDFS-14295.4.patch
          5 kB
          David Mollitor
        5. HDFS-14295.5.patch
          4 kB
          David Mollitor
        6. HDFS-14295.6.patch
          4 kB
          David Mollitor
        7. HDFS-14295.7.patch
          4 kB
          David Mollitor
        8. HDFS-14295.8.patch
          4 kB
          David Mollitor
        9. HDFS-14295.9.patch
          4 kB
          David Mollitor
        10. HDFS-14295.10.patch
          4 kB
          David Mollitor

          Issue Links

            Activity

              People

              • Assignee:
                belugabehr David Mollitor
                Reporter:
                belugabehr David Mollitor
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: