Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7213

processIncrementalBlockReport performance degradation

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager#processIncrementalBlockReport has a debug line that is missing a isDebugEnabled check. The write lock is being held. Coupled with the increase in incremental block reports from receiving blocks, under heavy load this log line noticeably degrades performance.

      1. HDFS-7213.1412804753
        1 kB
        Eric Payne
      2. HDFS-7213.1412806496.txt
        1.0 kB
        Eric Payne

        Activity

        Hide
        eepayne Eric Payne added a comment -

        Daryn Sharp, would you mind taking a look at this patch? Thanks.

        Show
        eepayne Eric Payne added a comment - Daryn Sharp , would you mind taking a look at this patch? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12673718/HDFS-7213.1412804753
        against trunk revision 2217e2f.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8355//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673718/HDFS-7213.1412804753 against trunk revision 2217e2f. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8355//console This message is automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Sorry, bad patch. This one should be better.

        Show
        eepayne Eric Payne added a comment - Sorry, bad patch. This one should be better.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12673726/HDFS-7213.1412806496.txt
        against trunk revision 2217e2f.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//artifact/patchprocess/patchReleaseAuditProblems.txt
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673726/HDFS-7213.1412806496.txt against trunk revision 2217e2f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8358//console This message is automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        Since the change is only adding an if debug around the log statement, unit testing would be impractical.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
        org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication

        I believe that the failure of TestDNFencingWithReplication is an unrelated issue. Please see HDFS-7221.

        Show
        eepayne Eric Payne added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Since the change is only adding an if debug around the log statement, unit testing would be impractical. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication I believe that the failure of TestDNFencingWithReplication is an unrelated issue. Please see HDFS-7221 .
        Hide
        kihwal Kihwal Lee added a comment -

        +1

        Show
        kihwal Kihwal Lee added a comment - +1
        Hide
        kihwal Kihwal Lee added a comment -

        Committed to trunk and cherry-picked to branch-2. Thanks for fixing it, Eric.

        Show
        kihwal Kihwal Lee added a comment - Committed to trunk and cherry-picked to branch-2. Thanks for fixing it, Eric.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6371 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6371/)
        HDFS-7213. processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6371 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6371/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #727 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/727/)
        HDFS-7213. processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #727 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/727/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1941 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1941/)
        HDFS-7213. processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1941 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1941/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        eepayne Eric Payne added a comment -

        Thanks Kihwal Lee

        Show
        eepayne Eric Payne added a comment - Thanks Kihwal Lee
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1916/)
        HDFS-7213. processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1916/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. (kihwal: rev e226b5b40d716b6d363c43a8783766b72734e347) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        mingma Ming Ma added a comment -

        For the 2.6.1 effort, this jira can be cleanly backported to branch-2.6.

        Show
        mingma Ming Ma added a comment - For the 2.6.1 effort, this jira can be cleanly backported to branch-2.6.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Cherry-picked to 2.6.1.

        Show
        vinayrpet Vinayakumar B added a comment - Cherry-picked to 2.6.1.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8298 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8298/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8298 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8298/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/)
        HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/ ) HDFS-7213 . processIncrementalBlockReport performance degradation. Contributed by Eric Payne. (vinayakumarb: rev d25cb8fe12d00faf3e8f3bfd23fd1b01981a340f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment - - edited

        This wasn't originally in 2.6.1, must have been committed to 2.6, which was already 2.6.2. I just committed this to 2.6.1 taking Sangjin Lee's cherry-pick, which must have come from branch-2.6.

        Ran compilation before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - - edited This wasn't originally in 2.6.1, must have been committed to 2.6, which was already 2.6.2. I just committed this to 2.6.1 taking Sangjin Lee 's cherry-pick, which must have come from branch-2.6. Ran compilation before the push.

          People

          • Assignee:
            eepayne Eric Payne
            Reporter:
            daryn Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development