Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8522

Change heavily recorded NN logs from INFO to DEBUG level

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      More specifically, the default namenode log settings have its log flooded with the following entries. This JIRA is opened to change them from INFO to DEBUG level.

       
      FSNamesystem.java:listCorruptFileBlocks 
      
      1. HDFS-8522.00.patch
        6 kB
        Xiaoyu Yao
      2. HDFS-8522.01.patch
        8 kB
        Xiaoyu Yao
      3. HDFS-8522.02.patch
        8 kB
        Xiaoyu Yao
      4. HDFS-8522.03.patch
        8 kB
        Xiaoyu Yao
      5. HDFS-8522.branch-2.00.patch
        7 kB
        Xiaoyu Yao
      6. HDFS-8522.branch-2.01.patch
        7 kB
        Xiaoyu Yao
      7. HDFS-8522.branch-2.7.00.patch
        11 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          xyao Xiaoyu Yao added a comment -

          RollingWindowManager.java:getTopUsersForMetric issue has been fixed in HDFS-7890.

          Show
          xyao Xiaoyu Yao added a comment - RollingWindowManager.java:getTopUsersForMetric issue has been fixed in HDFS-7890 .
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch that change the listCorruptFileBlocks log to DEBUG level and also clean up if (NameNode.stateChangeLog.isDebugEnabled()) as it is not needed for slf4j.

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch that change the listCorruptFileBlocks log to DEBUG level and also clean up if (NameNode.stateChangeLog.isDebugEnabled()) as it is not needed for slf4j.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Xiaoyu for doing the cleanup. Overall LGTM, but we should use string templating with braces for the stateChangeLog debug logs, to avoid string construction overhead.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Xiaoyu for doing the cleanup. Overall LGTM, but we should use string templating with braces for the stateChangeLog debug logs, to avoid string construction overhead.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 to Andrew's comment. Also does stateChangeLog use SLF4J? Not clear to me from a quick look.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 to Andrew's comment. Also does stateChangeLog use SLF4J? Not clear to me from a quick look.
          Hide
          andrew.wang Andrew Wang added a comment -

          It's using LoggerFactory.getLogger which is from org.slf4j so I think it's good. I think log4j will compile error when you pass all the extra parameters too.

          Show
          andrew.wang Andrew Wang added a comment - It's using LoggerFactory.getLogger which is from org.slf4j so I think it's good. I think log4j will compile error when you pass all the extra parameters too.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Andrew Wang and Arpit Agarwal for the review. I've updated the patch with string template for slf4j debug log in FSNamesystem.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Andrew Wang and Arpit Agarwal for the review. I've updated the patch with string template for slf4j debug log in FSNamesystem.
          Hide
          andrew.wang Andrew Wang added a comment -

          Some of those lines look a bit long, but checkstyle will catch it if so. +1 pending Jenkins.

          Show
          andrew.wang Andrew Wang added a comment - Some of those lines look a bit long, but checkstyle will catch it if so. +1 pending Jenkins.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fit the lines within 80 columns.

          Show
          xyao Xiaoyu Yao added a comment - Fit the lines within 80 columns.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          +1 for the .02 patch, pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited +1 for the .02 patch, pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 26s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 17s The applied patch generated 2 new checkstyle issues (total was 264, now 264).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 19s Pre-build of native portion
          -1 hdfs tests 160m 36s Tests failed in hadoop-hdfs.
              206m 34s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737989/HDFS-8522.02.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7588585
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11246/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11246/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11246/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11246/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 26s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 17s The applied patch generated 2 new checkstyle issues (total was 264, now 264). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 19s Pre-build of native portion -1 hdfs tests 160m 36s Tests failed in hadoop-hdfs.     206m 34s   Reason Tests Failed unit tests hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737989/HDFS-8522.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7588585 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11246/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11246/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11246/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11246/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          I don't add test because this is a log only changes. The 1st checkstyle issue is known and the 2nd one will be addressed by reverting unnecessary change at FSNamesystem.java:2513 for Log.Warn(...), which causes the line longer than 80 characters. The test failure is unrelated. The change is low risk, I will commit it shortly to trunk, branch-2 and branch-2.7.

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 7,628 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:2513: Line is longer than 80 characters (found 82).
          
          Show
          xyao Xiaoyu Yao added a comment - I don't add test because this is a log only changes. The 1st checkstyle issue is known and the 2nd one will be addressed by reverting unnecessary change at FSNamesystem.java:2513 for Log.Warn(...), which causes the line longer than 80 characters. The test failure is unrelated. The change is low risk, I will commit it shortly to trunk, branch-2 and branch-2.7. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 7,628 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:2513: Line is longer than 80 characters (found 82).
          Hide
          xyao Xiaoyu Yao added a comment -

          The only delta from .02->.03 is to revert unnecessary change at FSNamesystem.java:2513 for Log.Warn(...), which causes the line longer than 80 characters.

          Show
          xyao Xiaoyu Yao added a comment - The only delta from .02->.03 is to revert unnecessary change at FSNamesystem.java:2513 for Log.Warn(...), which causes the line longer than 80 characters.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v3 patch, pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v3 patch, pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738073/HDFS-8522.03.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3841d09
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11251/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738073/HDFS-8522.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3841d09 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11251/console This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7980 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7980/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7980 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7980/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch for branch-2.7, please review. Thanks!

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch for branch-2.7, please review. Thanks!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2.7 patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2.7 patch.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for reviewing the branch-2.7 patch.
          Attach the branch-2 patch. Please review.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for reviewing the branch-2.7 patch. Attach the branch-2 patch. Please review.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The truncate log level is info in the branch-2 patch. It's debug in trunk/branch-2.7. +1 with that fixed.

          Thanks for fixing this Xiaoyu Yao.

          Show
          arpitagarwal Arpit Agarwal added a comment - The truncate log level is info in the branch-2 patch. It's debug in trunk/branch-2.7. +1 with that fixed. Thanks for fixing this Xiaoyu Yao .
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal again for the review. branch-2.01.patch addresses the issue.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal again for the review. branch-2.01.patch addresses the issue.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1

          Show
          arpitagarwal Arpit Agarwal added a comment - +1
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks all for the review. I've committed the changes to trunk, branch-2 and branch-2.7.

          Show
          xyao Xiaoyu Yao added a comment - Thanks all for the review. I've committed the changes to trunk, branch-2 and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/220/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/950/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #950 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/950/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2148/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/209/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/218/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/)
          HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2166/ ) HDFS-8522 . Change heavily recorded NN logs from INFO to DEBUG level. (Contributed by Xiaoyu Yao) (xyao: rev 3841d09765bab332c9ae4803c5981799585b1f41) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development