Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9839

Reduce verbosity of processReport logging

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager#processReport logs one line for each invalidated block at INFO. HDFS-7503 moved this logging outside the NameSystem write lock but we still see the NameNode being slowed down when the number of block invalidations is very large e.g. just after a large amount of data is deleted.

            for (Block b : invalidatedBlocks) {
              blockLog.info("BLOCK* processReport: {} on node {} size {} does not " +
                  "belong to any file", b, node, b.getNumBytes());
            }
      

      We can change this statement to DEBUG and just log the number of block invalidations at INFO.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user arp7 opened a pull request:

        https://github.com/apache/hadoop/pull/78

        HDFS-9839. Reduce verbosity of processReport logging

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/arp7/hadoop HDFS-9839

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/hadoop/pull/78.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #78


        commit 8b23b41ada23168fe2cb71f4a3b920c68e66ee74
        Author: Arpit Agarwal <arp@apache.org>
        Date: 2016-02-20T18:43:14Z

        HDFS-9839. Reduce verbosity of processReport logging


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user arp7 opened a pull request: https://github.com/apache/hadoop/pull/78 HDFS-9839 . Reduce verbosity of processReport logging You can merge this pull request into a Git repository by running: $ git pull https://github.com/arp7/hadoop HDFS-9839 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/78.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #78 commit 8b23b41ada23168fe2cb71f4a3b920c68e66ee74 Author: Arpit Agarwal <arp@apache.org> Date: 2016-02-20T18:43:14Z HDFS-9839 . Reduce verbosity of processReport logging
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 14s trunk passed
        +1 compile 0m 51s trunk passed with JDK v1.8.0_72
        +1 compile 0m 43s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 1m 16s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 49s the patch passed with JDK v1.8.0_72
        +1 javac 0m 49s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_95
        +1 javac 0m 40s the patch passed
        +1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 151 unchanged - 1 fixed = 151 total (was 152)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95
        -1 unit 76m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 71m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        175m 21s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.tracing.TestTracing
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.mover.TestStorageMover
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Issue HDFS-9839
        GITHUB PR https://github.com/apache/hadoop/pull/78
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 796baa5082c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e54cc29
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14549/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14549/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 14s trunk passed +1 compile 0m 51s trunk passed with JDK v1.8.0_72 +1 compile 0m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 49s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_72 +1 javac 0m 49s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_95 +1 javac 0m 40s the patch passed +1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 151 unchanged - 1 fixed = 151 total (was 152) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95 -1 unit 76m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 71m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 175m 21s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.mover.TestStorageMover JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Issue HDFS-9839 GITHUB PR https://github.com/apache/hadoop/pull/78 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 796baa5082c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e54cc29 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14549/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14549/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14549/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Patch LGTM. +1.

        Show
        xyao Xiaoyu Yao added a comment - Patch LGTM. +1.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/hadoop/pull/78

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/78
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thank you for the review Xiaoyu Yao.

        I committed this to trunk, branch-2 and branch-2.8.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thank you for the review Xiaoyu Yao . I committed this to trunk, branch-2 and branch-2.8.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9335 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9335/)
        HDFS-9839. Reduce verbosity of processReport logging. (Contributed by (arp: rev d5abd293a890a8a1da48a166a291ae1c5644ad57)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9335 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9335/ ) HDFS-9839 . Reduce verbosity of processReport logging. (Contributed by (arp: rev d5abd293a890a8a1da48a166a291ae1c5644ad57) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            arpitagarwal Arpit Agarwal
            Reporter:
            arpitagarwal Arpit Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development