Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13564

PreAllocator for DfsClientShm

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.2
    • Fix Version/s: 3.0.2
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:

      Description

      When we do a stress test against Short-Circuit Local Reads, and found a bottleneck that allocating new DfsClientShm blocks a lot of slot allocatings on it.

      Currently, there are 128 slots per shm which means at most, 128 reads could be blocked by the shm allocation. Especially when stressed, the domain socket communication to datanode gets slow, and datanode could also have GC, which could cause some hundreds ms to allocate 1 shm, in turn, the reads. This is bad for some latency sensitive service, like Hbase.

      I'm working on the prototype and will upload the code and test result later. 

        Attachments

          Activity

            People

            • Assignee:
              leosun08 Lisheng Sun
              Reporter:
              xiegang112 Gang Xie
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: