Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9521

TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, TransferFsImage.receiveFile is logging total transfer time as below:

      double xferSec = Math.max(
             ((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);    
      long xferKb = received / 1024;
      LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb / xferSec))
      

      This is really useful, but it just measures the total method execution time, which includes time taken to download the image and do an fsync to all the namenode metadata directories.

      Sometime when troubleshooting these imager transfer problems, it's interesting to know which part of the process is being the bottleneck (whether network or disk write).

      This patch accounts time for image download and fsync to each disk separately, logging how much time did it take on each operation.

        Attachments

        1. HDFS-9521.004.patch
          4 kB
          Harsh J
        2. HDFS-9521-3.patch
          4 kB
          Wellington Chevreuil
        3. HDFS-9521-2.patch
          4 kB
          Wellington Chevreuil
        4. HDFS-9521.patch.1
          5 kB
          Wellington Chevreuil
        5. HDFS-9521.patch
          3 kB
          Wellington Chevreuil

          Activity

            People

            • Assignee:
              wchevreuil Wellington Chevreuil
              Reporter:
              wchevreuil Wellington Chevreuil
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: