Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6110

adding more slow action log in critical write path

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs:

      dfs.client.slow.io.warning.threshold.ms (Default 30 seconds)
      dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)
      Show
      Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs: dfs.client.slow.io.warning.threshold.ms (Default 30 seconds) dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)

      Description

      After digging a HBase write spike issue caused by slow buffer io in our cluster, just realize we'd better to add more abnormal latency warning log in write flow, such that if other guys hit HLog sync spike, we could know more detail info from HDFS side at the same time.
      Patch will be uploaded soon.

      1. HDFS-6110.txt
        6 kB
        Liang Xie
      2. HDFS-6110-v2.txt
        10 kB
        Liang Xie
      3. HDFS-6110v3.txt
        11 kB
        stack
      4. HDFS-6110v4.txt
        11 kB
        Liang Xie
      5. HDFS-6110v5.txt
        11 kB
        Liang Xie
      6. HDFS-6110v6.txt
        13 kB
        stack

        Issue Links

          Activity

          Liang Xie created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-3590 [ HDFS-3590 ]
          Todd Lipcon made changes -
          Link This issue relates to HDFS-3751 [ HDFS-3751 ]
          Liang Xie made changes -
          Attachment HDFS-6110.txt [ 12635246 ]
          Liang Xie made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Liang Xie made changes -
          Attachment HDFS-6110-v2.txt [ 12635457 ]
          Liang Xie made changes -
          Attachment HDFS-6110-v2.txt [ 12635457 ]
          Liang Xie made changes -
          Attachment HDFS-6110-v2.txt [ 12635463 ]
          stack made changes -
          Attachment HDFS-6110v3.txt [ 12637343 ]
          Liang Xie made changes -
          Attachment HDFS-6110v4.txt [ 12638010 ]
          Liang Xie made changes -
          Attachment HDFS-6110v5.txt [ 12641882 ]
          Liang Xie made changes -
          Attachment HDFS-6110v5.txt [ 12641882 ]
          Liang Xie made changes -
          Attachment HDFS-6110v5.txt [ 12641883 ]
          stack made changes -
          Attachment HDFS-6110v6.txt [ 12642004 ]
          stack made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs:

          dfs.client.slow.io.warning.threshold.ms (Default 30 seconds)
          dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)
          Fix Version/s 2.5.0 [ 12326264 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Liang Xie
              Reporter:
              Liang Xie
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development