Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4047

ClientRMService getApplications has high scheduler lock contention

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The getApplications call can be particuarly expensive because the code can call checkAccess on every application being tracked by the RM. checkAccess will often call scheduler.checkAccess which will grab the big scheduler lock. This can cause a lot of contention with the scheduler thread which is busy trying to process node heartbeats, app allocation requests, etc.

        Activity

        Hide
        jlowe Jason Lowe added a comment -

        In OOZIE-1729 Oozie started calling getApplications to look for applications with specific tags. This significantly increases the utilization of this method on a cluster that makes heavy use of Oozie.

        One quick fix for the Oozie use-case may be to swap the filter order. Rather than doing the expensive checkAccess call first, we can do all the other filtering first and finally verify the user has access before adding the app to the response. In the Oozie scenario most apps will be filtered by the tag check before we ever get to the checkAccess call.

        Show
        jlowe Jason Lowe added a comment - In OOZIE-1729 Oozie started calling getApplications to look for applications with specific tags. This significantly increases the utilization of this method on a cluster that makes heavy use of Oozie. One quick fix for the Oozie use-case may be to swap the filter order. Rather than doing the expensive checkAccess call first, we can do all the other filtering first and finally verify the user has access before adding the app to the response. In the Oozie scenario most apps will be filtered by the tag check before we ever get to the checkAccess call.
        Hide
        jlowe Jason Lowe added a comment -

        Patch that performs the checkAccess filter last rather than first.

        Show
        jlowe Jason Lowe added a comment - Patch that performs the checkAccess filter last rather than first.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 47s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 1s There were no new javac warning messages.
        +1 javadoc 9m 59s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 50s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 22s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 53m 27s Tests failed in hadoop-yarn-server-resourcemanager.
            92m 53s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          hadoop.yarn.server.resourcemanager.TestRMAdminService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12749935/YARN-4047.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7c796fd
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8824/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8824/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8824/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 53m 27s Tests failed in hadoop-yarn-server-resourcemanager.     92m 53s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749935/YARN-4047.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7c796fd hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8824/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8824/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8824/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        lgtm +1

        Show
        jianhe Jian He added a comment - lgtm +1
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.7. thanks Jason !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.7. thanks Jason !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8296 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8296/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8296 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8296/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #287 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/287/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1017 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1017/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #284 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/284/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2233 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2233/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2214 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2214/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/)
        YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/276/ ) YARN-4047 . ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java hadoop-yarn-project/CHANGES.txt
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.6.1. The patch applies cleanly. Ran compilation before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. The patch applies cleanly. Ran compilation before the push.

          People

          • Assignee:
            jlowe Jason Lowe
            Reporter:
            jlowe Jason Lowe
          • Votes:
            1 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development