Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6110

adding more slow action log in critical write path

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.5.0
    • datanode
    • None
    • Reviewed
    • Hide
      Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs:

      dfs.client.slow.io.warning.threshold.ms (Default 30 seconds)
      dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)
      Show
      Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs: dfs.client.slow.io.warning.threshold.ms (Default 30 seconds) dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)

    Description

      After digging a HBase write spike issue caused by slow buffer io in our cluster, just realize we'd better to add more abnormal latency warning log in write flow, such that if other guys hit HLog sync spike, we could know more detail info from HDFS side at the same time.
      Patch will be uploaded soon.

      Attachments

        1. HDFS-6110.txt
          6 kB
          Liang Xie
        2. HDFS-6110-v2.txt
          10 kB
          Liang Xie
        3. HDFS-6110v3.txt
          11 kB
          Michael Stack
        4. HDFS-6110v4.txt
          11 kB
          Liang Xie
        5. HDFS-6110v5.txt
          11 kB
          Liang Xie
        6. HDFS-6110v6.txt
          13 kB
          Michael Stack

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-8439 Adding more slow action log in critical read path

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3751 DN should log warnings for lengthy disk IOs

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3590 Print a WARN if the edit log sync period takes more than X time units

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              xieliang007 Liang Xie
              xieliang007 Liang Xie
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: