Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3170

Add more useful metrics for write latency

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the only write-latency related metric we expose is the total amount of time taken by opWriteBlock. This is practically useless, since (a) different blocks may be wildly different sizes, and (b) if the writer is only generating data slowly, it will make a block write take longer by no fault of the DN. I would like to propose two new metrics:
      1) flush-to-disk time: count how long it takes for each call to flush an incoming packet to disk (including the checksums). In most cases this will be close to 0, as it only flushes to buffer cache, but if the backing block device enters congested writeback, it can take much longer, which provides an interesting metric.
      2) round trip to downstream pipeline node: track the round trip latency for the part of the pipeline between the local node and its downstream neighbors. When we add a new packet to the ack queue, save the current timestamp. When we receive an ack, update the metric based on how long since we sent the original packet. This gives a metric of the total RTT through the pipeline. If we also include this metric in the ack to upstream, we can subtract the amount of time due to the later stages in the pipeline and have an accurate count of this particular link.

      1. hdfs-3170.txt
        16 kB
        Matthew Jacobs
      2. hdfs-3170.txt
        14 kB
        Matthew Jacobs
      3. hdfs-3170.txt
        12 kB
        Matthew Jacobs

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-3343 [ HDFS-3343 ]
          Todd Lipcon made changes -
          Link This issue is related to HDFS-942 [ HDFS-942 ]
          Matthew Jacobs made changes -
          Assignee Matthew Jacobs [ mjacobs ]
          Matthew Jacobs made changes -
          Attachment hdfs-3170.txt [ 12533205 ]
          Matthew Jacobs made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Matthew Jacobs made changes -
          Attachment hdfs-3170.txt [ 12534171 ]
          Matthew Jacobs made changes -
          Attachment hdfs-3170.txt [ 12535141 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Matthew Jacobs
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development