Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7258

CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0, 2.6.5
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      CacheReplicationMonitor rescan scheduler adds two INFO log entries every 30 seconds to HDSF NN log as shown below. This should be a DEBUG level log to avoid flooding the namenode log.

      2014-10-17 07:52:30,265 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:52:30,265 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:53:00,265 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:53:00,266 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
      2014-10-17 07:53:30,267 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:53:30,267 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:54:00,267 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:54:00,268 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:54:30,268 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:54:30,269 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:55:00,269 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:55:00,269 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
      2014-10-17 07:55:30,268 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:55:30,269 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:56:00,269 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:56:00,270 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:56:30,270 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
      2014-10-17 07:56:30,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
      2014-10-17 07:57:00,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:57:00,272 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
      2014-10-17 07:57:30,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:57:30,272 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
      2014-10-17 07:58:00,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
      2014-10-17 07:58:00,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
      2014-10-17 07:58:30,271 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds

      1. HDFS-7258.0.patch
        2 kB
        Xiaoyu Yao
      2. HDFS-7258.1.patch
        2 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          cmccabe Colin P. McCabe added a comment -

          Seems reasonable to me. We added those logs at INFO back when the mechanism was new, and never changed it.

          Show
          cmccabe Colin P. McCabe added a comment - Seems reasonable to me. We added those logs at INFO back when the mechanism was new, and never changed it.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hi Xiaoyu Yao, you don't need the "if" statements any more. slf4j is smart enough to not do any extra work if debug is not turned on. See http://javarevisited.blogspot.com/2013/08/why-use-sl4j-over-log4j-for-logging-in.html for details.

          So instead of this:

          182	              if (LOG.isDebugEnabled()) {
          183	                LOG.debug("Rescanning after " + (curTimeMs - startTimeMs) +
          184	                  " milliseconds");
          185	              }
          

          You just need this:

          LOG.debug("Rescanning after {} milliseconds", curTimeMs - startTimeMs);
          

          Similar comment for the LOG.debug("Scanned " + scannedDirectives ... log statement.

          Show
          cmccabe Colin P. McCabe added a comment - Hi Xiaoyu Yao , you don't need the "if" statements any more. slf4j is smart enough to not do any extra work if debug is not turned on. See http://javarevisited.blogspot.com/2013/08/why-use-sl4j-over-log4j-for-logging-in.html for details. So instead of this: 182 if (LOG.isDebugEnabled()) { 183 LOG.debug( "Rescanning after " + (curTimeMs - startTimeMs) + 184 " milliseconds" ); 185 } You just need this: LOG.debug( "Rescanning after {} milliseconds" , curTimeMs - startTimeMs); Similar comment for the LOG.debug("Scanned " + scannedDirectives ... log statement.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676694/HDFS-7258.0.patch
          against trunk revision 86cad00.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8504//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8504//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676694/HDFS-7258.0.patch against trunk revision 86cad00. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8504//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8504//console This message is automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Colin P. McCabe for reviewing and pointing me the slf4j blog. Patch updated.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Colin P. McCabe for reviewing and pointing me the slf4j blog. Patch updated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676814/HDFS-7258.1.patch
          against trunk revision 5b56ac4.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8514//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8514//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676814/HDFS-7258.1.patch against trunk revision 5b56ac4. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8514//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8514//console This message is automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          +1. I'll commit it shortly.

          Show
          wheat9 Haohui Mai added a comment - +1. I'll commit it shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6332 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6332/)
          HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6332 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6332/ ) HDFS-7258 . CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          wheat9 Haohui Mai added a comment -

          Committed to trunk and branch-2. Thanks Colin P. McCabe for the reviews, and Xiaoyu Yao for the contribution.

          Xiaoyu Yao, I think that you can leave the fix version blank when creating the jira. The field should be filled in when the jira is committed.

          Show
          wheat9 Haohui Mai added a comment - Committed to trunk and branch-2. Thanks Colin P. McCabe for the reviews, and Xiaoyu Yao for the contribution. Xiaoyu Yao , I think that you can leave the fix version blank when creating the jira. The field should be filled in when the jira is committed.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #722 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/722/)
          HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #722 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/722/ ) HDFS-7258 . CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1911/)
          HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1911/ ) HDFS-7258 . CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1936 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1936/)
          HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1936 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1936/ ) HDFS-7258 . CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level. Contributed by Xiaoyu Yao. (wheat9: rev 0942c99eba12f6baf5609c9621cd07b09618a97e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Xiaoyu Yao and Haohui Mai.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Xiaoyu Yao and Haohui Mai .
          Hide
          shv Konstantin Shvachko added a comment -

          Committed to branch-2.6.

          Show
          shv Konstantin Shvachko added a comment - Committed to branch-2.6.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development