Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2975

FSLeafQueue app lists are accessed without required locks

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      YARN-2910 adds explicit locked access to runnable and non-runnable apps in FSLeafQueue. As FSLeafQueue has getters for these, they can be accessed without locks in other places.

      1. yarn-2975-3.patch
        26 kB
        Karthik Kambatla
      2. yarn-2975-2.patch
        26 kB
        Karthik Kambatla
      3. yarn-2975-1.patch
        27 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          kasha Karthik Kambatla added a comment -

          Patch that removes methods that return the lists as is and replaces their use with other (new) methods.

          Show
          kasha Karthik Kambatla added a comment - Patch that removes methods that return the lists as is and replaces their use with other (new) methods.
          Hide
          rchiang Ray Chiang added a comment -

          For removeNonRunnableApp(), isRunnableApp(), isNonRunnableApp(), shouldn't the return value be in the finally block instead of the try block?

          Show
          rchiang Ray Chiang added a comment - For removeNonRunnableApp(), isRunnableApp(), isNonRunnableApp(), shouldn't the return value be in the finally block instead of the try block?
          Hide
          rchiang Ray Chiang added a comment -

          Same with getNumRunnableApps(), getNumNonRunnableApps(),

          Show
          rchiang Ray Chiang added a comment - Same with getNumRunnableApps(), getNumNonRunnableApps(),
          Hide
          kasha Karthik Kambatla added a comment -

          It shouldn't matter either way. I would prefer to leave it as is, or return after the finally block.

          Show
          kasha Karthik Kambatla added a comment - It shouldn't matter either way. I would prefer to leave it as is, or return after the finally block.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12687832/yarn-2975-1.patch
          against trunk revision f2d150e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 14 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6133//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6133//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6133//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687832/yarn-2975-1.patch against trunk revision f2d150e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 14 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6133//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6133//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6133//console This message is automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          None of the findbugs warnings are related to this patch.

          Show
          kasha Karthik Kambatla added a comment - None of the findbugs warnings are related to this patch.
          Hide
          rchiang Ray Chiang added a comment -

          +1 (non-binding). Code looks fine to me. Both unit tests run fine in my tree.

          Show
          rchiang Ray Chiang added a comment - +1 (non-binding). Code looks fine to me. Both unit tests run fine in my tree.
          Hide
          adhoot Anubhav Dhoot added a comment -

          The removeApp from one operation is now 2 steps. Just to confirm my understanding, are we relying on the locking in FairScheduler's calling methods (removeApplicationAttempt etc) to ensure consistency
          boolean wasRunnable = queue.isRunnableApp(attempt);
          queue.removeApp(attempt);

          Nit:
          suggest resetPreemptedResources -> resetPreemptedResourcesRunnableApps
          clearPreemptedResources -> clear PreemptedResourcesRunnableApps

          Show
          adhoot Anubhav Dhoot added a comment - The removeApp from one operation is now 2 steps. Just to confirm my understanding, are we relying on the locking in FairScheduler's calling methods (removeApplicationAttempt etc) to ensure consistency boolean wasRunnable = queue.isRunnableApp(attempt); queue.removeApp(attempt); Nit: suggest resetPreemptedResources -> resetPreemptedResourcesRunnableApps clearPreemptedResources -> clear PreemptedResourcesRunnableApps
          Hide
          kasha Karthik Kambatla added a comment -

          The removeApp from one operation is now 2 steps.

          removeApp did not expose the two operations outside, but had them.

          IIUC, an app can only go from non-runnable to runnable, but not vice-versa. For instance, see the following snippet in FairScheduler#executeMove. So, don't think we need to worry about consistency.

              if (wasRunnable && !nowRunnable) {
                throw new IllegalStateException("Should have already verified that app "
                    + attempt.getApplicationId() + " would be runnable in new queue");
              }
          

          Nit:
          suggest resetPreemptedResources -> resetPreemptedResourcesRunnableApps
          clearPreemptedResources -> clear PreemptedResourcesRunnableApps

          I thought of this, but decided against it. The additional RunnableApps doesn't add anything but extra characters.

          Show
          kasha Karthik Kambatla added a comment - The removeApp from one operation is now 2 steps. removeApp did not expose the two operations outside, but had them. IIUC, an app can only go from non-runnable to runnable, but not vice-versa. For instance, see the following snippet in FairScheduler#executeMove. So, don't think we need to worry about consistency. if (wasRunnable && !nowRunnable) { throw new IllegalStateException( "Should have already verified that app " + attempt.getApplicationId() + " would be runnable in new queue" ); } Nit: suggest resetPreemptedResources -> resetPreemptedResourcesRunnableApps clearPreemptedResources -> clear PreemptedResourcesRunnableApps I thought of this, but decided against it. The additional RunnableApps doesn't add anything but extra characters.
          Hide
          kasha Karthik Kambatla added a comment -

          Anubhav Dhoot - I see your point about consistency. If the app was not runnable at the time of check but becomes runnable by the time we remove it, we can miss untracking it. May be, we should clean up how we store runnability of an app in FSAppAttempt, and add transitions like rest of RM code to handle addition, move, removal of the apps. The scope might be too big to do in this JIRA. Okay with doing it in a follow-up?

          By the way, thanks a bunch for taking the time to review.

          Show
          kasha Karthik Kambatla added a comment - Anubhav Dhoot - I see your point about consistency. If the app was not runnable at the time of check but becomes runnable by the time we remove it, we can miss untracking it. May be, we should clean up how we store runnability of an app in FSAppAttempt, and add transitions like rest of RM code to handle addition, move, removal of the apps. The scope might be too big to do in this JIRA. Okay with doing it in a follow-up? By the way, thanks a bunch for taking the time to review.
          Hide
          adhoot Anubhav Dhoot added a comment -

          Yes I am worried about getting it wrong for maxRunningEnforcer. Before the change, we would inside a lock achieve the removal of the app whether it was in runnable or not and be reasonably sure.
          Now the splitting it into 2 non atomic steps outside as i listed above, and also 2 steps inside

           return removeRunnableApp(app) || removeNonRunnableApp(app) 

          , we might make it worse as each one leaves the lock before the other acquires. The application could be completely missed when it moves from nonrunnable to runnable in between.
          How about making removeApp do try to remove from both runnable or nonRunnable inside a single writelock. We can try removing the redundancies with removeRunnableApp and removeNonRunnableApp by having a fourth internal method that all 3 delegate via flags to limit where to look for the app.

          Show
          adhoot Anubhav Dhoot added a comment - Yes I am worried about getting it wrong for maxRunningEnforcer. Before the change, we would inside a lock achieve the removal of the app whether it was in runnable or not and be reasonably sure. Now the splitting it into 2 non atomic steps outside as i listed above, and also 2 steps inside return removeRunnableApp(app) || removeNonRunnableApp(app) , we might make it worse as each one leaves the lock before the other acquires. The application could be completely missed when it moves from nonrunnable to runnable in between. How about making removeApp do try to remove from both runnable or nonRunnable inside a single writelock. We can try removing the redundancies with removeRunnableApp and removeNonRunnableApp by having a fourth internal method that all 3 delegate via flags to limit where to look for the app.
          Hide
          kasha Karthik Kambatla added a comment -

          I understand the concern. Pre-YARN-2910, we were in the exact same place and haven't seen any issues in practice. Let me take a look again and see if we can do what you are suggesting - do all of removeApp while holding the writeLock. Initially, I had a single method, but had to split based on prior accesses.

          Show
          kasha Karthik Kambatla added a comment - I understand the concern. Pre- YARN-2910 , we were in the exact same place and haven't seen any issues in practice. Let me take a look again and see if we can do what you are suggesting - do all of removeApp while holding the writeLock. Initially, I had a single method, but had to split based on prior accesses.
          Hide
          kasha Karthik Kambatla added a comment -

          Updated patch to preserve behavior of FSLeafQueue#removeApp and add FSLeafQueue#removeNonRunnableApp separately.

          Show
          kasha Karthik Kambatla added a comment - Updated patch to preserve behavior of FSLeafQueue#removeApp and add FSLeafQueue#removeNonRunnableApp separately.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12688412/yarn-2975-2.patch
          against trunk revision d9e4d67.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 14 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6157//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6157//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6157//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12688412/yarn-2975-2.patch against trunk revision d9e4d67. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 14 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6157//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6157//console This message is automatically generated.
          Hide
          adhoot Anubhav Dhoot added a comment -

          Minor comment:
          The following comment might be misleading. One may assume this means the app will be removed regardless and the boolean return is only to indicate whether it happened to be nonRunnable

            /**
             * @return true if the app was non-runnable, false otherwise
             */
           public boolean removeNonRunnableApp(FSAppAttempt app) {
          

          LGTM otherwise

          Show
          adhoot Anubhav Dhoot added a comment - Minor comment: The following comment might be misleading. One may assume this means the app will be removed regardless and the boolean return is only to indicate whether it happened to be nonRunnable /** * @return true if the app was non-runnable, false otherwise */ public boolean removeNonRunnableApp(FSAppAttempt app) { LGTM otherwise
          Hide
          rkanter Robert Kanter added a comment -

          +1 after clarifying the comment that Anubhav pointed out

          Show
          rkanter Robert Kanter added a comment - +1 after clarifying the comment that Anubhav pointed out
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks Anubhav. Updated the comment to be clearer.

          The test failures and findbugs warnings look unrelated.

          Show
          kasha Karthik Kambatla added a comment - Thanks Anubhav. Updated the comment to be clearer. The test failures and findbugs warnings look unrelated.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for the review, Robert. I ll go ahead and commit this, if Jenkins doesn't complain of any new issues.

          Show
          kasha Karthik Kambatla added a comment - Thanks for the review, Robert. I ll go ahead and commit this, if Jenkins doesn't complain of any new issues.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12688455/yarn-2975-3.patch
          against trunk revision 390a7c1.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 15 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6160//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6160//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6160//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12688455/yarn-2975-3.patch against trunk revision 390a7c1. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 15 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6160//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6160//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6160//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6769 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6769/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6769 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6769/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/CHANGES.txt
          Hide
          kasha Karthik Kambatla added a comment -

          Committed this to trunk and branch-2 earlier today.

          Show
          kasha Karthik Kambatla added a comment - Committed this to trunk and branch-2 earlier today.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #48 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/48/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #48 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/48/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #782 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/782/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #782 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/782/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1980/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1980/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #45 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/45/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #45 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/45/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #49 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/49/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #49 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/49/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1999 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1999/)
          YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1999 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1999/ ) YARN-2975 . FSLeafQueue app lists are accessed without required locks. (kasha) (kasha: rev 24ee9e3431d27811530ffa01d8d241133fd643fe) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestMaxRunningAppsEnforcer.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerLeafQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Hide
          sjlee0 Sangjin Lee added a comment -

          I think we should backport this to branch-2.6. This is a very important follow-up fix to YARN-2910. Karthik Kambatla?

          Show
          sjlee0 Sangjin Lee added a comment - I think we should backport this to branch-2.6. This is a very important follow-up fix to YARN-2910 . Karthik Kambatla ?
          Hide
          kasha Karthik Kambatla added a comment -

          Makes sense. Sangjin Lee - want to do the honors?

          Show
          kasha Karthik Kambatla added a comment - Makes sense. Sangjin Lee - want to do the honors?
          Hide
          sjlee0 Sangjin Lee added a comment -

          I cherry-picked the fix to branch-2.6 (2.6.4).

          Show
          sjlee0 Sangjin Lee added a comment - I cherry-picked the fix to branch-2.6 (2.6.4).
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9060 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9060/)
          Add YARN-2975, YARN-3893, YARN-2902 and YARN-4354 to Release 2.6.4 entry (junping_du: rev b6c9d3fab9c76b03abd664858f64a4ebf3c2bb20)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9060 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9060/ ) Add YARN-2975 , YARN-3893 , YARN-2902 and YARN-4354 to Release 2.6.4 entry (junping_du: rev b6c9d3fab9c76b03abd664858f64a4ebf3c2bb20) hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              kasha Karthik Kambatla
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development