Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1471

The SLS simulator is not running the preemption policy for CapacityScheduler

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha1, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. This is because the policy needs to interact with a CapacityScheduler, and the wrapping done by the simulator breaks this.

      1. SLSCapacityScheduler.java
        6 kB
        Alejandro Abdelnur
      2. YARN-1471.2.patch
        44 kB
        Chris Douglas
      3. YARN-1471.patch
        40 kB
        Chris Douglas
      4. YARN-1471.patch
        40 kB
        Carlo Curino
      5. YARN-1471-branch-2.7.4.patch
        43 kB
        Ye Zhou
      6. YARN-1471-branch-2.8.patch
        44 kB
        Ye Zhou
      7. YARN-1471-branch-2.patch
        43 kB
        Ye Zhou

        Issue Links

          Activity

          Hide
          curino Carlo Curino added a comment -

          As I raised during the discussion in https://issues.apache.org/jira/browse/YARN-1021?focusedCommentId=13745799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745799 the simulator is not currently compatible with the CapacityScheduler preemption mechanisms.

          More generally the "wrapping" done is likely to break interactions among components that are accessing not only the top-level interfaces, but leverage specifics of each scheduler. It would be nice to find some better wrapping mechanism. Wei Yan, Alejandro Abdelnur... any idea?

          Show
          curino Carlo Curino added a comment - As I raised during the discussion in https://issues.apache.org/jira/browse/YARN-1021?focusedCommentId=13745799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745799 the simulator is not currently compatible with the CapacityScheduler preemption mechanisms. More generally the "wrapping" done is likely to break interactions among components that are accessing not only the top-level interfaces, but leverage specifics of each scheduler. It would be nice to find some better wrapping mechanism. Wei Yan , Alejandro Abdelnur ... any idea?
          Hide
          ywskycn Wei Yan added a comment -

          Thanks, Carlo Curino.
          I'll take a look at ProportionalCapacityPreemptionPolicy firstly.

          Show
          ywskycn Wei Yan added a comment - Thanks, Carlo Curino . I'll take a look at ProportionalCapacityPreemptionPolicy firstly.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Carlo Curino, I'm not familiar on how the ProportionalCapacityPreemptionPolicy gets hold of data from the CapacityScheduler. The SLS simply wraps the Scheduler implementation using a proxy pattern. Thus, the scheduler API is fully exposed even if wrapped. If the ProportionalCapacityPreemptionPolicy cast the Scheduler interface to the CapacityScheduler class, I would be inclined to say that this is not a SLS issue but a ProportionalCapacityPreemptionPolicy issue. Are the methods of the CapacityScheduler used by the ProportionalCapacityPreemptionPolicy general purpose to qualify being in the Scheduler API? If not, how about implementing a 'safety valve', something like public T <T> getComponent(Class<T> klass) in the Scheduler API, and the contract is to return NULL if such component is not implemented by the scheduler. Would something like this work?

          Show
          tucu00 Alejandro Abdelnur added a comment - Carlo Curino , I'm not familiar on how the ProportionalCapacityPreemptionPolicy gets hold of data from the CapacityScheduler. The SLS simply wraps the Scheduler implementation using a proxy pattern. Thus, the scheduler API is fully exposed even if wrapped. If the ProportionalCapacityPreemptionPolicy cast the Scheduler interface to the CapacityScheduler class, I would be inclined to say that this is not a SLS issue but a ProportionalCapacityPreemptionPolicy issue. Are the methods of the CapacityScheduler used by the ProportionalCapacityPreemptionPolicy general purpose to qualify being in the Scheduler API? If not, how about implementing a 'safety valve', something like public T <T> getComponent(Class<T> klass) in the Scheduler API, and the contract is to return NULL if such component is not implemented by the scheduler. Would something like this work?
          Hide
          curino Carlo Curino added a comment -

          The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific to the CapacityScheduler.

          The three offending methods are:

          • scheduler.getResourceCalculator();
          • scheduler.getRootQueue(); (which is a CSQueue)
          • scheduler.getClusterResources()

          However more generally this policy only makes sense when paired with the CapacityScheduler, so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler.
          This is coming up again in the work towards YARN-1051, as we have other "monitors" interacting with the underlying scheduler, possibly in ways that are specific to one specific scheduler.

          Speaking with Chris Douglas we seem to agree that the simulator would be much more useful if it allows to simulate specifics of each scheduler more than just exercise the general API (specifically I would not use it otherwise), and hacking the Scheduler API (and classes using it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems a job for aspect programming... though it comes with its own set of headaches (see mumak).

          Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler to those (assuming the rest of your metrics are collected in methods not used by the monitor).

          Show
          curino Carlo Curino added a comment - The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific to the CapacityScheduler. The three offending methods are: scheduler.getResourceCalculator(); scheduler.getRootQueue(); (which is a CSQueue) scheduler.getClusterResources() However more generally this policy only makes sense when paired with the CapacityScheduler, so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler. This is coming up again in the work towards YARN-1051 , as we have other "monitors" interacting with the underlying scheduler, possibly in ways that are specific to one specific scheduler. Speaking with Chris Douglas we seem to agree that the simulator would be much more useful if it allows to simulate specifics of each scheduler more than just exercise the general API (specifically I would not use it otherwise), and hacking the Scheduler API (and classes using it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems a job for aspect programming... though it comes with its own set of headaches (see mumak). Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler to those (assuming the rest of your metrics are collected in methods not used by the monitor).
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          As the ProportionalCapacityPreemptionPolicy is CS specific, why does not live within the CS itself? Shouldn't be that way?

          Show
          tucu00 Alejandro Abdelnur added a comment - As the ProportionalCapacityPreemptionPolicy is CS specific, why does not live within the CS itself? Shouldn't be that way?
          Hide
          curino Carlo Curino added a comment -

          We spoke with Alejandro, and a plausible solution goes as follows:
          The simulator instead of wrapping the scheduler sub-class it (and dynamically pick the right one, based on configs). The subclassing allows things like the ProportionalCapacityPreemptionPolicy (and future monitors) to work happily, and allows us to override only the methods relevant to the simulator.
          Wei, if you have nothing against it, I might try to give this a try.

          Show
          curino Carlo Curino added a comment - We spoke with Alejandro, and a plausible solution goes as follows: The simulator instead of wrapping the scheduler sub-class it (and dynamically pick the right one, based on configs). The subclassing allows things like the ProportionalCapacityPreemptionPolicy (and future monitors) to work happily, and allows us to override only the methods relevant to the simulator. Wei, if you have nothing against it, I might try to give this a try.
          Hide
          ywskycn Wei Yan added a comment -

          Sure. Thanks, Carlo Curino.
          Just let me know if I can help.

          Show
          ywskycn Wei Yan added a comment - Sure. Thanks, Carlo Curino . Just let me know if I can help.
          Hide
          curino Carlo Curino added a comment -

          Wei, I posted a patch that implements the change we discussed with Alejandro. Most of it is rather trivial, with one ugly exception. The subclassing of CapacityScheduler is basically a 1:1 copy of your ResourceSchedulerWrapper (changed to be a subclass instead of a wrapper). I don't have in mind a good alternative and I need to move foward with this somewhat quickly, if you have a better idea feel free to evolve our patch (or have a separate JIRA to prettify this).
          More importantly I tested it with both CapacityScheduler and FairScheduler and seems to work as intended, but I have limited understanding of all the metrics you collect and the functionalities of the simulator, as I just started using it, if you have a chance it would be great for you to test this more carefully.
          Thanks!

          Show
          curino Carlo Curino added a comment - Wei, I posted a patch that implements the change we discussed with Alejandro. Most of it is rather trivial, with one ugly exception. The subclassing of CapacityScheduler is basically a 1:1 copy of your ResourceSchedulerWrapper (changed to be a subclass instead of a wrapper). I don't have in mind a good alternative and I need to move foward with this somewhat quickly, if you have a better idea feel free to evolve our patch (or have a separate JIRA to prettify this). More importantly I tested it with both CapacityScheduler and FairScheduler and seems to work as intended, but I have limited understanding of all the metrics you collect and the functionalities of the simulator, as I just started using it, if you have a chance it would be great for you to test this more carefully. Thanks!
          Hide
          curino Carlo Curino added a comment -

          Patch implementing proposed change.

          Show
          curino Carlo Curino added a comment - Patch implementing proposed change.
          Hide
          ywskycn Wei Yan added a comment -

          Thanks, Carlo Curino. I'll test it locally.

          Show
          ywskycn Wei Yan added a comment - Thanks, Carlo Curino . I'll test it locally.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Carlo Curino, how about the attached alternate impl for the SLSCapacityScheduler.java? it fully leverages the existing wrapper without duplicating code. You'll need to add a new constructor the the wrapper to take a ResourceScheduler as parameter.

          The SchedulerWrapper interface should be name SLSSchedulerWrapper and the implementation SLSSchedulerWrapperImpl.

          HTH

          Show
          tucu00 Alejandro Abdelnur added a comment - Carlo Curino , how about the attached alternate impl for the SLSCapacityScheduler.java? it fully leverages the existing wrapper without duplicating code. You'll need to add a new constructor the the wrapper to take a ResourceScheduler as parameter. The SchedulerWrapper interface should be name SLSSchedulerWrapper and the implementation SLSSchedulerWrapperImpl. HTH
          Hide
          curino Carlo Curino added a comment -

          Mmh... The problem is that you are creating a cycle passing "this" as the scheduler. E.g., calls to reinitialize for example will cycle between the two classes forever ) I confirmed this integrating your code with the rest.

          Show
          curino Carlo Curino added a comment - Mmh... The problem is that you are creating a cycle passing "this" as the scheduler. E.g., calls to reinitialize for example will cycle between the two classes forever ) I confirmed this integrating your code with the rest.
          Hide
          curino Carlo Curino added a comment -

          Alejandro, I tried the second version of what you proposed. It does not crash and seem to sort of work, but talking with Chris it has some very bad properties. If the inner CapacityScheduler (now or in the future) invokes from within one of its methods any of the methods you guys want to trap, this implementation would completely miss it.

          This is also true for the ResourceWrapper more generally I think, so probably the trick of subclassing instead of wrapping would be safer to propagate to the FairScheduler side as well. The maintenance cost here is that when a new scheduler is added, you have to create another one of this sub classes, but this seems rare enough of an event to be ok.

          Show
          curino Carlo Curino added a comment - Alejandro, I tried the second version of what you proposed. It does not crash and seem to sort of work, but talking with Chris it has some very bad properties. If the inner CapacityScheduler (now or in the future) invokes from within one of its methods any of the methods you guys want to trap, this implementation would completely miss it. This is also true for the ResourceWrapper more generally I think, so probably the trick of subclassing instead of wrapping would be safer to propagate to the FairScheduler side as well. The maintenance cost here is that when a new scheduler is added, you have to create another one of this sub classes, but this seems rare enough of an event to be ok.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12617451/SLSCapacityScheduler.java
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2686//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617451/SLSCapacityScheduler.java against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2686//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619212/YARN-1471.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2687//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619212/YARN-1471.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2687//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619430/YARN-1471.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-sls.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2694//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2694//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619430/YARN-1471.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-sls. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2694//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2694//console This message is automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          After discussing with Wei Yan and Alejandro Abdelnur, the current workaround is sufficient. A more extensive refactoring- including tests to ensure the CS continues to work- will be deferred to YARN-1518.

          Show
          chris.douglas Chris Douglas added a comment - After discussing with Wei Yan and Alejandro Abdelnur , the current workaround is sufficient. A more extensive refactoring- including tests to ensure the CS continues to work- will be deferred to YARN-1518 .
          Hide
          chris.douglas Chris Douglas added a comment -

          I committed this. Thanks Carlo

          Show
          chris.douglas Chris Douglas added a comment - I committed this. Thanks Carlo
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4914 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4914/)
          YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
          Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4914 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4914/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #426 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/426/)
          YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
          Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #426 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/426/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1643 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1643/)
          YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
          Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1643 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1643/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1617/)
          YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
          Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1617/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
          Hide
          curino Carlo Curino added a comment -

          Thanks Chris! and thanks to Wei and Alejandro for useful conversations.

          Show
          curino Carlo Curino added a comment - Thanks Chris! and thanks to Wei and Alejandro for useful conversations.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/)
          Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/)
          Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/)
          Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
          Hide
          zhouyejoe Ye Zhou added a comment -

          Konstantin Shvachko Attached patch for 2.7 branch. Build and Tests passed locally. Trigger Jenkins.

          Show
          zhouyejoe Ye Zhou added a comment - Konstantin Shvachko Attached patch for 2.7 branch. Build and Tests passed locally. Trigger Jenkins.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey Ye Zhou or Chris Douglas do you mind updating path for branch-2 as well. I tried to merge but it didn't work.
          Cannot commit to 2.7 without committing to 2.

          Show
          shv Konstantin Shvachko added a comment - Hey Ye Zhou or Chris Douglas do you mind updating path for branch-2 as well. I tried to merge but it didn't work. Cannot commit to 2.7 without committing to 2.
          Hide
          chris.douglas Chris Douglas added a comment -

          Carlo Curino is this contained in YARN-6608? If so, maybe we should look into backporting that, instead of individual SLS patches.

          Show
          chris.douglas Chris Douglas added a comment - Carlo Curino is this contained in YARN-6608 ? If so, maybe we should look into backporting that, instead of individual SLS patches.
          Hide
          curino Carlo Curino added a comment -

          I believe it should be. YARN-6608 will need some more massaging, but it shouldn't be too much work.

          Show
          curino Carlo Curino added a comment - I believe it should be. YARN-6608 will need some more massaging, but it shouldn't be too much work.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey guys. Let's target full SLS feature for the next release. Would be too big change for a dot one.
          Here I just need a rebase of the branch-2 patch, which has been attached, but got stale, and which is just sufficient for our needs in 2.7. I am pretty confident about this one in particular as we were running it internally in production. Also since this is in trunk it may ease the backport effort of YARN-6608.

          Show
          shv Konstantin Shvachko added a comment - Hey guys. Let's target full SLS feature for the next release. Would be too big change for a dot one. Here I just need a rebase of the branch-2 patch, which has been attached, but got stale, and which is just sufficient for our needs in 2.7. I am pretty confident about this one in particular as we were running it internally in production. Also since this is in trunk it may ease the backport effort of YARN-6608 .
          Hide
          shv Konstantin Shvachko added a comment -

          I see a new patch in YARN-6608. So this is progressing. And if this change is incorporated there we just need to port into branch-2.7. Will do just that if there are no objections.

          Show
          shv Konstantin Shvachko added a comment - I see a new patch in YARN-6608 . So this is progressing. And if this change is incorporated there we just need to port into branch-2.7. Will do just that if there are no objections.
          Hide
          zhouyejoe Ye Zhou added a comment -

          Konstantin ShvachkoPatches for branch 2.7, branch 2.8 and branch 2 build and pass unit tests locally.

          Show
          zhouyejoe Ye Zhou added a comment - Konstantin Shvachko Patches for branch 2.7, branch 2.8 and branch 2 build and pass unit tests locally.
          Hide
          shv Konstantin Shvachko added a comment -

          I double-checked all SLS tests pass.
          Just committed this to branch-2, -2.8, and -2.7. Thank you Ye Zhou for backports.

          Show
          shv Konstantin Shvachko added a comment - I double-checked all SLS tests pass. Just committed this to branch-2, -2.8, and -2.7. Thank you Ye Zhou for backports.

            People

            • Assignee:
              curino Carlo Curino
              Reporter:
              curino Carlo Curino
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development