Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1471

The SLS simulator is not running the preemption policy for CapacityScheduler

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. This is because the policy needs to interact with a CapacityScheduler, and the wrapping done by the simulator breaks this.

      1. SLSCapacityScheduler.java
        6 kB
        Alejandro Abdelnur
      2. YARN-1471.2.patch
        44 kB
        Chris Douglas
      3. YARN-1471.patch
        40 kB
        Chris Douglas
      4. YARN-1471.patch
        40 kB
        Carlo Curino
      5. YARN-1471-branch-2.7.4.patch
        44 kB
        Ye Zhou

        Activity

        Hide
        curino Carlo Curino added a comment -

        As I raised during the discussion in https://issues.apache.org/jira/browse/YARN-1021?focusedCommentId=13745799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745799 the simulator is not currently compatible with the CapacityScheduler preemption mechanisms.

        More generally the "wrapping" done is likely to break interactions among components that are accessing not only the top-level interfaces, but leverage specifics of each scheduler. It would be nice to find some better wrapping mechanism. Wei Yan, Alejandro Abdelnur... any idea?

        Show
        curino Carlo Curino added a comment - As I raised during the discussion in https://issues.apache.org/jira/browse/YARN-1021?focusedCommentId=13745799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745799 the simulator is not currently compatible with the CapacityScheduler preemption mechanisms. More generally the "wrapping" done is likely to break interactions among components that are accessing not only the top-level interfaces, but leverage specifics of each scheduler. It would be nice to find some better wrapping mechanism. Wei Yan , Alejandro Abdelnur ... any idea?
        Hide
        ywskycn Wei Yan added a comment -

        Thanks, Carlo Curino.
        I'll take a look at ProportionalCapacityPreemptionPolicy firstly.

        Show
        ywskycn Wei Yan added a comment - Thanks, Carlo Curino . I'll take a look at ProportionalCapacityPreemptionPolicy firstly.
        Hide
        tucu00 Alejandro Abdelnur added a comment -

        Carlo Curino, I'm not familiar on how the ProportionalCapacityPreemptionPolicy gets hold of data from the CapacityScheduler. The SLS simply wraps the Scheduler implementation using a proxy pattern. Thus, the scheduler API is fully exposed even if wrapped. If the ProportionalCapacityPreemptionPolicy cast the Scheduler interface to the CapacityScheduler class, I would be inclined to say that this is not a SLS issue but a ProportionalCapacityPreemptionPolicy issue. Are the methods of the CapacityScheduler used by the ProportionalCapacityPreemptionPolicy general purpose to qualify being in the Scheduler API? If not, how about implementing a 'safety valve', something like public T <T> getComponent(Class<T> klass) in the Scheduler API, and the contract is to return NULL if such component is not implemented by the scheduler. Would something like this work?

        Show
        tucu00 Alejandro Abdelnur added a comment - Carlo Curino , I'm not familiar on how the ProportionalCapacityPreemptionPolicy gets hold of data from the CapacityScheduler. The SLS simply wraps the Scheduler implementation using a proxy pattern. Thus, the scheduler API is fully exposed even if wrapped. If the ProportionalCapacityPreemptionPolicy cast the Scheduler interface to the CapacityScheduler class, I would be inclined to say that this is not a SLS issue but a ProportionalCapacityPreemptionPolicy issue. Are the methods of the CapacityScheduler used by the ProportionalCapacityPreemptionPolicy general purpose to qualify being in the Scheduler API? If not, how about implementing a 'safety valve', something like public T <T> getComponent(Class<T> klass) in the Scheduler API, and the contract is to return NULL if such component is not implemented by the scheduler. Would something like this work?
        Hide
        curino Carlo Curino added a comment -

        The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific to the CapacityScheduler.

        The three offending methods are:

        • scheduler.getResourceCalculator();
        • scheduler.getRootQueue(); (which is a CSQueue)
        • scheduler.getClusterResources()

        However more generally this policy only makes sense when paired with the CapacityScheduler, so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler.
        This is coming up again in the work towards YARN-1051, as we have other "monitors" interacting with the underlying scheduler, possibly in ways that are specific to one specific scheduler.

        Speaking with Chris Douglas we seem to agree that the simulator would be much more useful if it allows to simulate specifics of each scheduler more than just exercise the general API (specifically I would not use it otherwise), and hacking the Scheduler API (and classes using it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems a job for aspect programming... though it comes with its own set of headaches (see mumak).

        Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler to those (assuming the rest of your metrics are collected in methods not used by the monitor).

        Show
        curino Carlo Curino added a comment - The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific to the CapacityScheduler. The three offending methods are: scheduler.getResourceCalculator(); scheduler.getRootQueue(); (which is a CSQueue) scheduler.getClusterResources() However more generally this policy only makes sense when paired with the CapacityScheduler, so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler. This is coming up again in the work towards YARN-1051 , as we have other "monitors" interacting with the underlying scheduler, possibly in ways that are specific to one specific scheduler. Speaking with Chris Douglas we seem to agree that the simulator would be much more useful if it allows to simulate specifics of each scheduler more than just exercise the general API (specifically I would not use it otherwise), and hacking the Scheduler API (and classes using it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems a job for aspect programming... though it comes with its own set of headaches (see mumak). Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler to those (assuming the rest of your metrics are collected in methods not used by the monitor).
        Hide
        tucu00 Alejandro Abdelnur added a comment -

        As the ProportionalCapacityPreemptionPolicy is CS specific, why does not live within the CS itself? Shouldn't be that way?

        Show
        tucu00 Alejandro Abdelnur added a comment - As the ProportionalCapacityPreemptionPolicy is CS specific, why does not live within the CS itself? Shouldn't be that way?
        Hide
        curino Carlo Curino added a comment -

        We spoke with Alejandro, and a plausible solution goes as follows:
        The simulator instead of wrapping the scheduler sub-class it (and dynamically pick the right one, based on configs). The subclassing allows things like the ProportionalCapacityPreemptionPolicy (and future monitors) to work happily, and allows us to override only the methods relevant to the simulator.
        Wei, if you have nothing against it, I might try to give this a try.

        Show
        curino Carlo Curino added a comment - We spoke with Alejandro, and a plausible solution goes as follows: The simulator instead of wrapping the scheduler sub-class it (and dynamically pick the right one, based on configs). The subclassing allows things like the ProportionalCapacityPreemptionPolicy (and future monitors) to work happily, and allows us to override only the methods relevant to the simulator. Wei, if you have nothing against it, I might try to give this a try.
        Hide
        ywskycn Wei Yan added a comment -

        Sure. Thanks, Carlo Curino.
        Just let me know if I can help.

        Show
        ywskycn Wei Yan added a comment - Sure. Thanks, Carlo Curino . Just let me know if I can help.
        Hide
        curino Carlo Curino added a comment -

        Wei, I posted a patch that implements the change we discussed with Alejandro. Most of it is rather trivial, with one ugly exception. The subclassing of CapacityScheduler is basically a 1:1 copy of your ResourceSchedulerWrapper (changed to be a subclass instead of a wrapper). I don't have in mind a good alternative and I need to move foward with this somewhat quickly, if you have a better idea feel free to evolve our patch (or have a separate JIRA to prettify this).
        More importantly I tested it with both CapacityScheduler and FairScheduler and seems to work as intended, but I have limited understanding of all the metrics you collect and the functionalities of the simulator, as I just started using it, if you have a chance it would be great for you to test this more carefully.
        Thanks!

        Show
        curino Carlo Curino added a comment - Wei, I posted a patch that implements the change we discussed with Alejandro. Most of it is rather trivial, with one ugly exception. The subclassing of CapacityScheduler is basically a 1:1 copy of your ResourceSchedulerWrapper (changed to be a subclass instead of a wrapper). I don't have in mind a good alternative and I need to move foward with this somewhat quickly, if you have a better idea feel free to evolve our patch (or have a separate JIRA to prettify this). More importantly I tested it with both CapacityScheduler and FairScheduler and seems to work as intended, but I have limited understanding of all the metrics you collect and the functionalities of the simulator, as I just started using it, if you have a chance it would be great for you to test this more carefully. Thanks!
        Hide
        curino Carlo Curino added a comment -

        Patch implementing proposed change.

        Show
        curino Carlo Curino added a comment - Patch implementing proposed change.
        Hide
        ywskycn Wei Yan added a comment -

        Thanks, Carlo Curino. I'll test it locally.

        Show
        ywskycn Wei Yan added a comment - Thanks, Carlo Curino . I'll test it locally.
        Hide
        tucu00 Alejandro Abdelnur added a comment -

        Carlo Curino, how about the attached alternate impl for the SLSCapacityScheduler.java? it fully leverages the existing wrapper without duplicating code. You'll need to add a new constructor the the wrapper to take a ResourceScheduler as parameter.

        The SchedulerWrapper interface should be name SLSSchedulerWrapper and the implementation SLSSchedulerWrapperImpl.

        HTH

        Show
        tucu00 Alejandro Abdelnur added a comment - Carlo Curino , how about the attached alternate impl for the SLSCapacityScheduler.java? it fully leverages the existing wrapper without duplicating code. You'll need to add a new constructor the the wrapper to take a ResourceScheduler as parameter. The SchedulerWrapper interface should be name SLSSchedulerWrapper and the implementation SLSSchedulerWrapperImpl. HTH
        Hide
        curino Carlo Curino added a comment -

        Mmh... The problem is that you are creating a cycle passing "this" as the scheduler. E.g., calls to reinitialize for example will cycle between the two classes forever ) I confirmed this integrating your code with the rest.

        Show
        curino Carlo Curino added a comment - Mmh... The problem is that you are creating a cycle passing "this" as the scheduler. E.g., calls to reinitialize for example will cycle between the two classes forever ) I confirmed this integrating your code with the rest.
        Hide
        curino Carlo Curino added a comment -

        Alejandro, I tried the second version of what you proposed. It does not crash and seem to sort of work, but talking with Chris it has some very bad properties. If the inner CapacityScheduler (now or in the future) invokes from within one of its methods any of the methods you guys want to trap, this implementation would completely miss it.

        This is also true for the ResourceWrapper more generally I think, so probably the trick of subclassing instead of wrapping would be safer to propagate to the FairScheduler side as well. The maintenance cost here is that when a new scheduler is added, you have to create another one of this sub classes, but this seems rare enough of an event to be ok.

        Show
        curino Carlo Curino added a comment - Alejandro, I tried the second version of what you proposed. It does not crash and seem to sort of work, but talking with Chris it has some very bad properties. If the inner CapacityScheduler (now or in the future) invokes from within one of its methods any of the methods you guys want to trap, this implementation would completely miss it. This is also true for the ResourceWrapper more generally I think, so probably the trick of subclassing instead of wrapping would be safer to propagate to the FairScheduler side as well. The maintenance cost here is that when a new scheduler is added, you have to create another one of this sub classes, but this seems rare enough of an event to be ok.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617451/SLSCapacityScheduler.java
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2686//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617451/SLSCapacityScheduler.java against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2686//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12619212/YARN-1471.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2687//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619212/YARN-1471.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2687//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12619430/YARN-1471.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-sls.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2694//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2694//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619430/YARN-1471.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-sls. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2694//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2694//console This message is automatically generated.
        Hide
        chris.douglas Chris Douglas added a comment -

        After discussing with Wei Yan and Alejandro Abdelnur, the current workaround is sufficient. A more extensive refactoring- including tests to ensure the CS continues to work- will be deferred to YARN-1518.

        Show
        chris.douglas Chris Douglas added a comment - After discussing with Wei Yan and Alejandro Abdelnur , the current workaround is sufficient. A more extensive refactoring- including tests to ensure the CS continues to work- will be deferred to YARN-1518 .
        Hide
        chris.douglas Chris Douglas added a comment -

        I committed this. Thanks Carlo

        Show
        chris.douglas Chris Douglas added a comment - I committed this. Thanks Carlo
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4914 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4914/)
        YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
        Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4914 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4914/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #426 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/426/)
        YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
        Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #426 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/426/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1643 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1643/)
        YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
        Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1643 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1643/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1617/)
        YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
        Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173)

        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml
        • /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1617/ ) YARN-1471 . Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173 ) /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml
        Hide
        curino Carlo Curino added a comment -

        Thanks Chris! and thanks to Wei and Alejandro for useful conversations.

        Show
        curino Carlo Curino added a comment - Thanks Chris! and thanks to Wei and Alejandro for useful conversations.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/)
        Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/)
        Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/)
        Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed)

        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/ ) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) hadoop-yarn-project/CHANGES.txt
        Hide
        zhouyejoe Ye Zhou added a comment -

        Konstantin Shvachko Attached patch for 2.7 branch. Build and Tests passed locally. Trigger Jenkins.

        Show
        zhouyejoe Ye Zhou added a comment - Konstantin Shvachko Attached patch for 2.7 branch. Build and Tests passed locally. Trigger Jenkins.
        Hide
        shv Konstantin Shvachko added a comment -

        Hey Ye Zhou or Chris Douglas do you mind updating path for branch-2 as well. I tried to merge but it didn't work.
        Cannot commit to 2.7 without committing to 2.

        Show
        shv Konstantin Shvachko added a comment - Hey Ye Zhou or Chris Douglas do you mind updating path for branch-2 as well. I tried to merge but it didn't work. Cannot commit to 2.7 without committing to 2.

          People

          • Assignee:
            curino Carlo Curino
            Reporter:
            curino Carlo Curino
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development