Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-707

Provide a jobconf property for explicitly assigning a job to a pool

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      add mapred.fairscheduler.pool property to define which pool a job belongs to.

      Description

      A common use case of the fair scheduler is to have one pool per user, but then to define some special pools for various production jobs, import jobs, etc. Therefore, it would be nice if jobs went by default to the pool of the user who submitted them, but there was a setting to explicitly place a job in another pool. Today, this can be achieved through a sort of trick in the JobConf:

      <property>
        <name>mapred.fairscheduler.poolnameproperty</name>
        <value>pool.name</value>
      </property>
      
      <property>
        <name>pool.name</name>
        <value>${user.name}</value>
      </property>
      

      This JIRA proposes to add a property called mapred.fairscheduler.pool that allows a job to be placed directly into a pool, avoiding the need for this trick.

      1. MAPREDUCE-707-2-apache.patch
        8 kB
        Alan Heirich
      2. MAPREDUCE-707-1-apache.patch
        8 kB
        Alan Heirich
      3. MAPREDUCE-707-apache.patch
        6 kB
        Alan Heirich

        Activity

        Hide
        Alan Heirich added a comment -

        I think the right steps are:
        Create a new property mapred.fairscheduler.pool with default value user.name
        Change default value of mapred.fairscheduler.poolnameproperty to mapred.fairscheduler.pool
        Add a unit test

        Can anyone tell me where to find the initialization of these properties?

        Show
        Alan Heirich added a comment - I think the right steps are: Create a new property mapred.fairscheduler.pool with default value user.name Change default value of mapred.fairscheduler.poolnameproperty to mapred.fairscheduler.pool Add a unit test Can anyone tell me where to find the initialization of these properties?
        Hide
        Matei Zaharia added a comment -

        Hi Alan,

        If you just want to get this behaviour as a fair scheduler user, the properties can be set in your mapred-site.xml. If you want to submit a patch so that this behaviour is the default, you have to edit mapred-default.xml, which is available in src or perhaps src/java in the mapreduce project.

        Show
        Matei Zaharia added a comment - Hi Alan, If you just want to get this behaviour as a fair scheduler user, the properties can be set in your mapred-site.xml. If you want to submit a patch so that this behaviour is the default, you have to edit mapred-default.xml, which is available in src or perhaps src/java in the mapreduce project.
        Hide
        Matei Zaharia added a comment -

        I should also add that my original intent in the JIRA was to have a separate way of setting the pool that doesn't require using the mapred.fairscheduler.poolnameproperty. This way, users could set poolnameproperty to something else (e.g. group.name for a Unix group). However, I think it's overkill to add a second way of setting the pool name, so I'd be okay with just a patch to mapred-default.xml if that works.

        Show
        Matei Zaharia added a comment - I should also add that my original intent in the JIRA was to have a separate way of setting the pool that doesn't require using the mapred.fairscheduler.poolnameproperty. This way, users could set poolnameproperty to something else (e.g. group.name for a Unix group). However, I think it's overkill to add a second way of setting the pool name, so I'd be okay with just a patch to mapred-default.xml if that works.
        Hide
        dhruba borthakur added a comment -

        > I think it's overkill to add a second way of setting the pool name, so I'd be okay with just a patch to mapred-default.xml if that works.

        +1.

        Show
        dhruba borthakur added a comment - > I think it's overkill to add a second way of setting the pool name, so I'd be okay with just a patch to mapred-default.xml if that works. +1.
        Hide
        Alan Heirich added a comment -

        What about PoolManager.java initialize() which reads the value of mapred.fairscheduler.poolnameproperty? It looks like this enforces the default values in conf.get(). If I do something like this then do we need to patch mapred-default.xml?
        public void initialize() throws IOException, SAXException,
        AllocationConfigurationException, ParserConfigurationException {
        Configuration conf = scheduler.getConf();
        + this.pool = conf.get(
        + "mapred.fairscheduler.pool", JobContext.USER_NAME);
        this.poolNameProperty = conf.get(

        • "mapred.fairscheduler.poolnameproperty", JobContext.USER_NAME);
          + "mapred.fairscheduler.poolnameproperty", this.pool);
        Show
        Alan Heirich added a comment - What about PoolManager.java initialize() which reads the value of mapred.fairscheduler.poolnameproperty? It looks like this enforces the default values in conf.get(). If I do something like this then do we need to patch mapred-default.xml? public void initialize() throws IOException, SAXException, AllocationConfigurationException, ParserConfigurationException { Configuration conf = scheduler.getConf(); + this.pool = conf.get( + "mapred.fairscheduler.pool", JobContext.USER_NAME); this.poolNameProperty = conf.get( "mapred.fairscheduler.poolnameproperty", JobContext.USER_NAME); + "mapred.fairscheduler.poolnameproperty", this.pool);
        Hide
        Matei Zaharia added a comment -

        I don't think that does what you want it to do, because it just sets poolNameProperty to a different value. What you want is for each individual job's pool to be determined based on which properties are in its JobConf (if it has mapred.fairscheduler.pool, use that; otherwise use whatever property poolNameProperty is set to; and if that doesn't exist in the JobConf either, use DEFAULT_POOL_NAME). The XML code I posted makes the job's JobConf implement this logic, by having the "pool" property be whatever the user sets it to if the user provides a setting, and making it default to the value of user.name otherwise. But there's no way to achieve this same behavior by just changing PoolManager.initialize to set poolNameProperty another way.

        Having thought about this some more, I think the most elegant implementation is to actually change PoolManger.getPoolName(JobInProgress job) to explicitly check whether the job's JobConf has the key "mapred.fairscheduler.pool" set, and if so, return that value; otherwise, it should return conf.get(poolNameProperty) as it does now. This should be a simple change to PoolManager.getPoolName. You will also need to change PoolManager.setPool() to set the "mapred.fairscheduler.pool" property rather than the poolNameProperty property on the job. Let me know if this makes sense.

        Show
        Matei Zaharia added a comment - I don't think that does what you want it to do, because it just sets poolNameProperty to a different value. What you want is for each individual job's pool to be determined based on which properties are in its JobConf (if it has mapred.fairscheduler.pool, use that; otherwise use whatever property poolNameProperty is set to; and if that doesn't exist in the JobConf either, use DEFAULT_POOL_NAME). The XML code I posted makes the job's JobConf implement this logic, by having the "pool" property be whatever the user sets it to if the user provides a setting, and making it default to the value of user.name otherwise. But there's no way to achieve this same behavior by just changing PoolManager.initialize to set poolNameProperty another way. Having thought about this some more, I think the most elegant implementation is to actually change PoolManger.getPoolName(JobInProgress job) to explicitly check whether the job's JobConf has the key "mapred.fairscheduler.pool" set, and if so, return that value; otherwise, it should return conf.get(poolNameProperty) as it does now. This should be a simple change to PoolManager.getPoolName. You will also need to change PoolManager.setPool() to set the "mapred.fairscheduler.pool" property rather than the poolNameProperty property on the job. Let me know if this makes sense.
        Hide
        Hemanth Yamijala added a comment -

        Matei, maybe it makes more sense to add fairshare scheduler properties to a scheduler specific file rather than mapred-*.xml, no ? This way any framework level changes that handle the framework's configuration would not be tied up by specific scheduler implementation. And also vice versa. Let me know what you think.

        Show
        Hemanth Yamijala added a comment - Matei, maybe it makes more sense to add fairshare scheduler properties to a scheduler specific file rather than mapred-*.xml, no ? This way any framework level changes that handle the framework's configuration would not be tied up by specific scheduler implementation. And also vice versa. Let me know what you think.
        Hide
        Matei Zaharia added a comment -

        I agree, and the solution I proposed above doesn't require changing mapred-default.xml. Many fair scheduler properties are in fair-scheduler.xml, but it would be nice to move some of the other ones that are still in mapred-site.xml to there. This would also make it easier to toggle them at runtime.

        Show
        Matei Zaharia added a comment - I agree, and the solution I proposed above doesn't require changing mapred-default.xml. Many fair scheduler properties are in fair-scheduler.xml, but it would be nice to move some of the other ones that are still in mapred-site.xml to there. This would also make it easier to toggle them at runtime.
        Hide
        Alan Heirich added a comment -

        Sure Matei, what you suggest sounds really simple and I'm testing it now. Thanks.

        Show
        Alan Heirich added a comment - Sure Matei, what you suggest sounds really simple and I'm testing it now. Thanks.
        Hide
        Alan Heirich added a comment -

        I'm finding that the demand for the mapSchedulable and reduceSchedulable objects are not updating as a result of removing and adding jobs to a pool. As a result of this calls to PoolManager.setPool do not cause the pool demands to update. (setPool calls removeJob() and addJob()).

        I've written a unit test that submits jobs to pools, tries to change their pool, and checks getDemand() to verify the right thing happened. This test is failing because getDemand() shows no changes in the demand.

        Is this the expected behavior of getDemand()???

        Show
        Alan Heirich added a comment - I'm finding that the demand for the mapSchedulable and reduceSchedulable objects are not updating as a result of removing and adding jobs to a pool. As a result of this calls to PoolManager.setPool do not cause the pool demands to update. (setPool calls removeJob() and addJob()). I've written a unit test that submits jobs to pools, tries to change their pool, and checks getDemand() to verify the right thing happened. This test is failing because getDemand() shows no changes in the demand. Is this the expected behavior of getDemand()???
        Hide
        Matei Zaharia added a comment -

        Hi Alan,

        Demands are only updated when the fair scheduler's update() function is called (which calls updateDemand in turn). All the code that calls setPool calls scheduler.update() afterwards. So you should do that in the unit test too.

        Show
        Matei Zaharia added a comment - Hi Alan, Demands are only updated when the fair scheduler's update() function is called (which calls updateDemand in turn). All the code that calls setPool calls scheduler.update() afterwards. So you should do that in the unit test too.
        Hide
        Alan Heirich added a comment -

        I guess another way to put this is: if we need to call updateDemand() to keep the demand up to date, should setPool() call updateDemand() after changing the pool for a job?

        Show
        Alan Heirich added a comment - I guess another way to put this is: if we need to call updateDemand() to keep the demand up to date, should setPool() call updateDemand() after changing the pool for a job?
        Hide
        Matei Zaharia added a comment -

        The reason I haven't made the PoolManager methods call updateDemand is that FairScheduler.update() does other things as well, and doing updateDemand without doing a full update() could potentially break some of the algorithms. (I'm not sure that it does so right now, but it would have been a problem in earlier versions). Therefore, I wanted all the updates to always happen through FairScheduler.update(). I'd rather not make the PoolManager call update() all the time because it would be better if the PoolManager didn't have to be modified whenever the structure of FairScheduler changes. All of the other unit tests call update() too, so I think it's fine not to do it in setPool.

        Show
        Matei Zaharia added a comment - The reason I haven't made the PoolManager methods call updateDemand is that FairScheduler.update() does other things as well, and doing updateDemand without doing a full update() could potentially break some of the algorithms. (I'm not sure that it does so right now, but it would have been a problem in earlier versions). Therefore, I wanted all the updates to always happen through FairScheduler.update(). I'd rather not make the PoolManager call update() all the time because it would be better if the PoolManager didn't have to be modified whenever the structure of FairScheduler changes. All of the other unit tests call update() too, so I think it's fine not to do it in setPool.
        Hide
        Matei Zaharia added a comment -

        In other words, I want to treat PoolManager, PoolSchedulable, JobSchedulable, etc as data structures, and decide externally when updates need to happen and when they don't, so that all that control logic is in one or two places (the event handlers in the FairScheduler and the UI code in FairSchedulerServlet).

        Show
        Matei Zaharia added a comment - In other words, I want to treat PoolManager, PoolSchedulable, JobSchedulable, etc as data structures, and decide externally when updates need to happen and when they don't, so that all that control logic is in one or two places (the event handlers in the FairScheduler and the UI code in FairSchedulerServlet).
        Hide
        Alan Heirich added a comment -

        adds mapred.fairscheduler.pool property, use it to specify pool name

        Show
        Alan Heirich added a comment - adds mapred.fairscheduler.pool property, use it to specify pool name
        Hide
        Alan Heirich added a comment -

        I would like to request a code review of MAPREDUCE-707-apache.patch, it is intended to resolve this JIRA.

        Show
        Alan Heirich added a comment - I would like to request a code review of MAPREDUCE-707 -apache.patch, it is intended to resolve this JIRA.
        Hide
        Matei Zaharia added a comment -

        Here are some comments on the patch:

        1. Instead of using the string "mapred.fairscheduler.pool" in multiple places in PoolManager, make it a constant at the top of the file (something like EXPLICIT_POOL_PROPERTY).
        2. Add a comment to PoolManager.getPoolName to explain the logic (first look for the explicit pool property, then for the property named by poolNameProperty, and finally default to DEFAULT_POOL_NAME).
        3. Add a unit test for PoolManager.getPoolName that tries each of those cases (explicit property is set, no explicit property but poolNameProperty is used, or neither is used). Right now your existing unit test checks that setPool works but there's no test that submits a job with mapred.fairscheduler.pool directly.
        4. Instead of assertEquals(0, scheduler.getPoolManager().getPoolName(job2).compareTo("poolA")) you can probably use a version of assertEquals that works on strings.
        5. In the documentation, instead of saying "This property is ignored if mapred.fairscheduler.pool is specified." for the poolnameproperty, it would be better to say that the poolnameproperty is used only for jobs in which mapred.fairscheduler.pool is not explicitly set.
        Show
        Matei Zaharia added a comment - Here are some comments on the patch: Instead of using the string "mapred.fairscheduler.pool" in multiple places in PoolManager, make it a constant at the top of the file (something like EXPLICIT_POOL_PROPERTY). Add a comment to PoolManager.getPoolName to explain the logic (first look for the explicit pool property, then for the property named by poolNameProperty, and finally default to DEFAULT_POOL_NAME). Add a unit test for PoolManager.getPoolName that tries each of those cases (explicit property is set, no explicit property but poolNameProperty is used, or neither is used). Right now your existing unit test checks that setPool works but there's no test that submits a job with mapred.fairscheduler.pool directly. Instead of assertEquals(0, scheduler.getPoolManager().getPoolName(job2).compareTo("poolA")) you can probably use a version of assertEquals that works on strings. In the documentation, instead of saying "This property is ignored if mapred.fairscheduler.pool is specified." for the poolnameproperty, it would be better to say that the poolnameproperty is used only for jobs in which mapred.fairscheduler.pool is not explicitly set.
        Hide
        Alan Heirich added a comment -

        revised patch per review comments

        Show
        Alan Heirich added a comment - revised patch per review comments
        Hide
        Alan Heirich added a comment -

        A patch after incorporating changes suggested in the review comments.

        Show
        Alan Heirich added a comment - A patch after incorporating changes suggested in the review comments.
        Hide
        Alan Heirich added a comment -

        Please review MAPREDUCE-707-1-apache.patch. Thanks.

        Show
        Alan Heirich added a comment - Please review MAPREDUCE-707 -1-apache.patch. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12424051/MAPREDUCE-707-apache.patch
        against trunk revision 832362.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424051/MAPREDUCE-707-apache.patch against trunk revision 832362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/224/console This message is automatically generated.
        Hide
        Matei Zaharia added a comment -

        This looks pretty good, except that testPoolAssignment fails when I run the unit tests. I think the problem is with job4, where you set "mapred.fairscheduler.poolnameproperty" in the job's Configuration (jobConf2), not in the fair scheduler's configuration. You need to set the poolNameProperty when you create the fair scheduler object. That's what the code used to do with the POOL_PROPERTY string at the top, but you can't set the pool name property to mapred.fairscheduler.pool, because that wouldn't be testing anything. I'd suggest leaving the POOL_PROPERTY as "pool" and trying to set job4's pool through that.

        Also, for sanity, in job1 (where you set mapred.fairscheduler.pool directly), you should say the "pool" property to something other than poolA to make sure it isn't used.

        Finally, two small nitpicks:

        1. In the test line with assertEquals(scheduler.getPoolManager().getPoolName(job2), "poolA"), you should switch the two parameters (put "poolA" first); the first parameter is always supposed to be the value expected.
        2. Regarding the comment on getPoolName - the pool name property used by default is "user.name", not "project". I think I forgot to fix that comment a while back.
        Show
        Matei Zaharia added a comment - This looks pretty good, except that testPoolAssignment fails when I run the unit tests. I think the problem is with job4, where you set "mapred.fairscheduler.poolnameproperty" in the job's Configuration (jobConf2), not in the fair scheduler's configuration. You need to set the poolNameProperty when you create the fair scheduler object. That's what the code used to do with the POOL_PROPERTY string at the top, but you can't set the pool name property to mapred.fairscheduler.pool, because that wouldn't be testing anything. I'd suggest leaving the POOL_PROPERTY as "pool" and trying to set job4's pool through that. Also, for sanity, in job1 (where you set mapred.fairscheduler.pool directly), you should say the "pool" property to something other than poolA to make sure it isn't used. Finally, two small nitpicks: In the test line with assertEquals(scheduler.getPoolManager().getPoolName(job2), "poolA"), you should switch the two parameters (put "poolA" first); the first parameter is always supposed to be the value expected. Regarding the comment on getPoolName - the pool name property used by default is "user.name", not "project". I think I forgot to fix that comment a while back.
        Hide
        Matei Zaharia added a comment -

        By the way, here's a tip if you want to run the unit tests faster - you can run just the fair scheduler's unit test with ant -Dtestcase=TestFairScheduler test.

        Show
        Matei Zaharia added a comment - By the way, here's a tip if you want to run the unit tests faster - you can run just the fair scheduler's unit test with ant -Dtestcase=TestFairScheduler test.
        Hide
        Alan Heirich added a comment -

        further revisions per comments.

        Show
        Alan Heirich added a comment - further revisions per comments.
        Hide
        Alan Heirich added a comment -

        Oops - I thought TestFairScheduler would be run as part of "ant test". I guess not. It passes now.

        Hudson reported that TestGridmixSubmission failed, but that passes in my workspace. I'm on Mac OS X and I saw some tests fail from a fresh build that should have passed.

        Please see MAPREDUCE-707-2-apache.patch

        Show
        Alan Heirich added a comment - Oops - I thought TestFairScheduler would be run as part of "ant test". I guess not. It passes now. Hudson reported that TestGridmixSubmission failed, but that passes in my workspace. I'm on Mac OS X and I saw some tests fail from a fresh build that should have passed. Please see MAPREDUCE-707 -2-apache.patch
        Hide
        Alan Heirich added a comment -

        corrected patch

        Show
        Alan Heirich added a comment - corrected patch
        Hide
        Matei Zaharia added a comment -

        Thanks Alan, this looks good! +1 from me. I'll wait for the Hudson automated test and then commit it if there are no warnings.

        Show
        Matei Zaharia added a comment - Thanks Alan, this looks good! +1 from me. I'll wait for the Hudson automated test and then commit it if there are no warnings.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12424090/MAPREDUCE-707-2-apache.patch
        against trunk revision 832362.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424090/MAPREDUCE-707-2-apache.patch against trunk revision 832362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/225/console This message is automatically generated.
        Hide
        Alan Heirich added a comment -

        This Hudson result is frustrating. TestGridmixSubmission passes when I run it in my workspace on Mac OS X. I cannot reproduce the failure. But Hudson compains that testSubmit fails! The diagnostic is:
        Error Message

        Mismatched output bytes 1737801/1764471
        Stacktrace

        junit.framework.AssertionFailedError: Mismatched output bytes 1737801/1764471
        at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231)
        at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140)
        at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263)
        at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297

        Show
        Alan Heirich added a comment - This Hudson result is frustrating. TestGridmixSubmission passes when I run it in my workspace on Mac OS X. I cannot reproduce the failure. But Hudson compains that testSubmit fails! The diagnostic is: Error Message Mismatched output bytes 1737801/1764471 Stacktrace junit.framework.AssertionFailedError: Mismatched output bytes 1737801/1764471 at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297
        Hide
        Matei Zaharia added a comment -

        The test failure seems to be due to Hudson being flaky, because the test passes on my laptop too. Therefore I've committed the patch. Thanks Alan!

        Show
        Matei Zaharia added a comment - The test failure seems to be due to Hudson being flaky, because the test passes on my laptop too. Therefore I've committed the patch. Thanks Alan!
        Hide
        Matei Zaharia added a comment -

        Alan, you should edit the JIRA to assign it to yourself through the "edit this issue" link.

        Show
        Matei Zaharia added a comment - Alan, you should edit the JIRA to assign it to yourself through the "edit this issue" link.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #110 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/110/)
        . Provide a jobconf property for explicitly assigning a job to
        a pool in the Fair Scheduler. Contributed by Alan Heirich.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #110 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/110/ ) . Provide a jobconf property for explicitly assigning a job to a pool in the Fair Scheduler. Contributed by Alan Heirich.
        Hide
        Alan Heirich added a comment -

        I don't seem able to assign to myself, perhaps because the issue is closed. That's ok.

        Show
        Alan Heirich added a comment - I don't seem able to assign to myself, perhaps because the issue is closed. That's ok.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/136/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/136/ )

          People

          • Assignee:
            Alan Heirich
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development