Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14292

Introduce Java ExecutorService to DataXceiverServer

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      I wanted to investigate dfs.datanode.max.transfer.threads from hdfs-site.xml. It is described as "Specifies the maximum number of threads to use for transferring data in and out of the DN." The default value is 4096. I found it interesting because 4096 threads sounds like a lot to me. I'm not sure how a system with 8-16 cores would react to this large a thread count. Intuitively, I would say that the overhead of context switching would be immense.

      During my investigation, I discovered the following setup in the DataXceiverServer class:

      1. A peer connects to a DataNode
      2. A new thread is spun up to service this connection
      3. The thread runs to completion
      4. The tread dies

      It would perhaps be better if we used a thread pool to better manage the lifecycle of the service threads and to allow the DataNode to re-use existing threads, saving on the need to create and spin-up threads on demand.

      In this JIRA, I have added a couple of things:

      1. Added a thread pool to DataXceiverServer class that, on demand, will create up to dfs.datanode.max.transfer.threads. A thread that has completed its prior duties will stay idle for up to 60 seconds (configurable), it will be retired if no new work has arrived.
      2. Added new methods to the Peer Interface to allow for better logging and less code within each Thread (DataXceiver).
      3. Updated the Thread code (DataXceiver) regarding its interactions with blockReceiver instance variable

        Attachments

        1. HDFS-14292.1.patch
          37 kB
          David Mollitor
        2. HDFS-14292.2.patch
          41 kB
          David Mollitor
        3. HDFS-14292.3.patch
          41 kB
          David Mollitor
        4. HDFS-14292.4.patch
          48 kB
          David Mollitor
        5. HDFS-14292.5.patch
          51 kB
          David Mollitor
        6. HDFS-14292.6.patch
          51 kB
          David Mollitor
        7. HDFS-14292.7.patch
          57 kB
          David Mollitor
        8. HDFS-14292.8.patch
          61 kB
          David Mollitor
        9. HDFS-14292.8.patch
          61 kB
          David Mollitor

          Issue Links

            Activity

              People

              • Assignee:
                belugabehr David Mollitor
                Reporter:
                belugabehr David Mollitor
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: