Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2243

DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None
    • Environment:

      Using Fedora 14 on a quad core phenom system

      Description

      I am running the YCSB benchmark against HBase, sometimes against a single node, sometimes against a cluster of 6 systems. As the load increases into thousands of TPS, especially on the single node, I can see that the datanode runs very high system time and seems to be bottlenecked by how fast it can create the threads to handle the new connections in DataXceiverServer.run. By "perf top" I can see the process spends about 12% of all its time in pthread_create, and in hprof profiles I can see there are tens of thousands of threads created in just a few minutes of test execution.

      Does anyone else observe this bottleneck? Is there a major challenge to using a thread pool of DataXceivers in this situation?

        Attachments

        1. datanode-perf-110808.gif
          9 kB
          Eric Caspole
        2. HDFS-2234-branch-0.20-append.patch
          10 kB
          Eric Caspole
        3. HDFS-2243-0.23-110909.txt
          3 kB
          Eric Caspole
        4. HDFS-2243-0.23-110909.patch
          3 kB
          Eric Caspole

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ecaspole Eric Caspole
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: