Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3803

BlockPoolSliceScanner new work period notice is very chatty at INFO level

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 2.0.1-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Environment:

      Hadoop 2.0.1-alpha-SNAPSHOT

    • Hadoop Flags:
      Reviewed

      Description

      One line of ~140 chars logged every 5 seconds.

      1. HDFS-3803.patch
        0.9 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          Sorry for the bother, I didn't see HDFS-3194.

          Show
          Andrew Purtell added a comment - Sorry for the bother, I didn't see HDFS-3194 .
          Hide
          Eli Collins added a comment -

          Yea, now that we don't start a new period every 5 seconds it's no longer chatty even at info level.

          Show
          Eli Collins added a comment - Yea, now that we don't start a new period every 5 seconds it's no longer chatty even at info level.
          Hide
          Suresh Srinivas added a comment -

          Looks like HDFS-3194 turned this log back to INFO level.

          Show
          Suresh Srinivas added a comment - Looks like HDFS-3194 turned this log back to INFO level.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/)
          HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/ ) HDFS-3803 . Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/)
          HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/ ) HDFS-3803 . Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Suresh Srinivas added a comment -

          Uma, the log can be turned back to info in HDFS-3194 as well.

          Show
          Suresh Srinivas added a comment - Uma, the log can be turned back to info in HDFS-3194 as well.
          Hide
          Uma Maheswara Rao G added a comment -

          Linked this issue to HDFS-3194, reasonable to undo this once HDFS-3194 goes in.
          (Actually HDFS-3194 addressing one more potential issue, that will be bit serious than this. But observation of that issue comes from this logs. Most of the times we will run with INFO logs only. If we change that to DEBUG, we would have missed that actual potential issue...More details on HDFS-3194)
          Need to check, because of this change whether that patch going stale and need to regenerate again.

          Show
          Uma Maheswara Rao G added a comment - Linked this issue to HDFS-3194 , reasonable to undo this once HDFS-3194 goes in. (Actually HDFS-3194 addressing one more potential issue, that will be bit serious than this. But observation of that issue comes from this logs. Most of the times we will run with INFO logs only. If we change that to DEBUG, we would have missed that actual potential issue...More details on HDFS-3194 ) Need to check, because of this change whether that patch going stale and need to regenerate again.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2643/)
          HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2643/ ) HDFS-3803 . Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2578 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2578/)
          HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2578 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2578/ ) HDFS-3803 . Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2603 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2603/)
          HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2603 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2603/ ) HDFS-3803 . Change BlockPoolSliceScanner chatty INFO log to DEBUG. Contributed by Andrew Purtell (Revision 1373174) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373174 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Suresh Srinivas added a comment -

          I disagree, without the "Starting a new period" message it's very difficult to diagnose the block pool scanner in production scenarios.

          Let me know what production scenario you have in mind. Debug logs should be the last resort for this kind of debugging. For thread related issues one can look at java stack. I agree we should do it with metrics.

          Currently this log is an annoyance. When HDFS-3194 goes in we can undo this change.

          Show
          Suresh Srinivas added a comment - I disagree, without the "Starting a new period" message it's very difficult to diagnose the block pool scanner in production scenarios. Let me know what production scenario you have in mind. Debug logs should be the last resort for this kind of debugging. For thread related issues one can look at java stack. I agree we should do it with metrics. Currently this log is an annoyance. When HDFS-3194 goes in we can undo this change.
          Hide
          Andrew Purtell added a comment -

          If the default period is supposed to be 3 weeks, then this message wouldn't represent the large fraction of log (therefore low S/N) as current. Whatever.

          Show
          Andrew Purtell added a comment - If the default period is supposed to be 3 weeks, then this message wouldn't represent the large fraction of log (therefore low S/N) as current. Whatever.
          Hide
          Andy Isaacson added a comment -

          LOG.debug seems reasonable level for it.

          I disagree, without the "Starting a new period" message it's very difficult to diagnose the block pool scanner in production scenarios. Production cannot run with debug enabled due to other cases where debug logging is actually expensive. So making this critical status message a LOG.debug makes it effectively useless.

          I'd be OK with replacing all of these logs with a useful set of (for example) JMX metrics so that the scanner is more observable in an automatable/graphable fashion, but until that happens, this message needs to stay in at LOG.info.

          Show
          Andy Isaacson added a comment - LOG.debug seems reasonable level for it. I disagree, without the "Starting a new period" message it's very difficult to diagnose the block pool scanner in production scenarios. Production cannot run with debug enabled due to other cases where debug logging is actually expensive. So making this critical status message a LOG.debug makes it effectively useless. I'd be OK with replacing all of these logs with a useful set of (for example) JMX metrics so that the scanner is more observable in an automatable/graphable fashion, but until that happens, this message needs to stay in at LOG.info.
          Hide
          Suresh Srinivas added a comment -

          Please revert this change and review HDFS-3194.

          Not sure if you saw my previous comment. Debug level seems reasonable for that log. Do you disagree.

          I will get to HDFS-3194 as soon as I can.

          Show
          Suresh Srinivas added a comment - Please revert this change and review HDFS-3194 . Not sure if you saw my previous comment. Debug level seems reasonable for that log. Do you disagree. I will get to HDFS-3194 as soon as I can.
          Hide
          Andy Isaacson added a comment -

          Please revert this change and review HDFS-3194.

          Show
          Andy Isaacson added a comment - Please revert this change and review HDFS-3194 .
          Hide
          Suresh Srinivas added a comment -

          @Andy, I think LOG.debug seems reasonable level for it. I am not sure if this calls for a revert. I will look at HDFS-3194.

          Show
          Suresh Srinivas added a comment - @Andy, I think LOG.debug seems reasonable level for it. I am not sure if this calls for a revert. I will look at HDFS-3194 .
          Hide
          Suresh Srinivas added a comment -

          Committed to 2.1

          Show
          Suresh Srinivas added a comment - Committed to 2.1
          Hide
          Andy Isaacson added a comment -

          The BlockPoolScanner is supposed to be starting a new period every three weeks, not every 5 seconds. See HDFS-3194.

          I think this -LOG.info +LOG.debug change should be reverted and https://issues.apache.org/jira/browse/HDFS-3194?focusedCommentId=13399085&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13399085 should be merged instead.

          Show
          Andy Isaacson added a comment - The BlockPoolScanner is supposed to be starting a new period every three weeks, not every 5 seconds. See HDFS-3194 . I think this -LOG.info +LOG.debug change should be reverted and https://issues.apache.org/jira/browse/HDFS-3194?focusedCommentId=13399085&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13399085 should be merged instead.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Andrew.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Andrew.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch. I will commit it soon.

          Show
          Suresh Srinivas added a comment - +1 for the patch. I will commit it soon.
          Hide
          Andrew Purtell added a comment -

          Trivial patch applies to both trunk and branch-2.

          Show
          Andrew Purtell added a comment - Trivial patch applies to both trunk and branch-2.

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development