Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-931

rumen should use its own interpolation classes to create runtimes for simulated tasks

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, when a simulator or benchmark is running and simulating hadoop jobs using rumen data, and rumen's runtime system is used to get execution times for the tasks in the simulated jobs, rumen would use some ad hoc code, despite the fact that rumen has a perfectly good interpolation framework to generate random variables that fit discrete CDFs.

      We should use the interpolation framework.

      1. patch-931-b.patch
        13 kB
        Dick King
      2. MAPREDUCE-931--2009-09-16--1633.patch
        11 kB
        Dick King

        Activity

        Hide
        Dick King added a comment -

        There is no separate test case for this patch, which is a code cleanup. ZombieJob was introduced in MAPREDUCE-751's patch, but it had an ad hoc interpolation engine. We refactored it here to use the new interpolation engine. This can affect performance during simulations, because ZombieJob created a whole new local interpolation table every time it needed an interpolation, even if it's identical to the one that would have been created for another interpolation in the same job. We now only create one interpolator for each job.

        TestZombieJob and TestPiecewiseLinearInterpolation are test cases for this technology.

        Show
        Dick King added a comment - There is no separate test case for this patch, which is a code cleanup. ZombieJob was introduced in MAPREDUCE-751 's patch, but it had an ad hoc interpolation engine. We refactored it here to use the new interpolation engine. This can affect performance during simulations, because ZombieJob created a whole new local interpolation table every time it needed an interpolation, even if it's identical to the one that would have been created for another interpolation in the same job. We now only create one interpolator for each job. TestZombieJob and TestPiecewiseLinearInterpolation are test cases for this technology.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418191/patch-931.patch
        against trunk revision 808730.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/28/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418191/patch-931.patch against trunk revision 808730. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/28/console This message is automatically generated.
        Hide
        Dick King added a comment -

        I had referenced the previous patch to the wrong branch. There is no change in the code.

        Show
        Dick King added a comment - I had referenced the previous patch to the wrong branch. There is no change in the code.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418193/patch-931-a.patch
        against trunk revision 808730.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/29/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418193/patch-931-a.patch against trunk revision 808730. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/29/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418199/patch-931-b.patch
        against trunk revision 809843.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2228 javac compiler warnings (more than the trunk's current 2226 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418199/patch-931-b.patch against trunk revision 809843. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2228 javac compiler warnings (more than the trunk's current 2226 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/1/console This message is automatically generated.
        Hide
        Dick King added a comment -

        The core tests are those failed without the patch.

        I'll look into the new warnings Wednesday.

        Show
        Dick King added a comment - The core tests are those failed without the patch. I'll look into the new warnings Wednesday.
        Hide
        Dick King added a comment -

        This patch fixes a small bug in CDFRandomGenerator, and also makes ZombieJob use a CDFPiecewiseLinearRandomGenerator to do its interpolation from a CDF rather than the ad hoc code that was there. This is an important code cleanup because the old code built new tables for each CDF, and more importantly because it makes plugging in a new kind of interpolator in the future a one-line change.

        We did not add tests. The results are non-deterministic. The patch testing tool will complain that there is no new test case. I did check some output manually and it looked plausible.

        Show
        Dick King added a comment - This patch fixes a small bug in CDFRandomGenerator , and also makes ZombieJob use a CDFPiecewiseLinearRandomGenerator to do its interpolation from a CDF rather than the ad hoc code that was there. This is an important code cleanup because the old code built new tables for each CDF, and more importantly because it makes plugging in a new kind of interpolator in the future a one-line change. We did not add tests. The results are non-deterministic. The patch testing tool will complain that there is no new test case. I did check some output manually and it looked plausible.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419830/MAPREDUCE-931--2009-09-16--1633.patch
        against trunk revision 815628.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419830/MAPREDUCE-931--2009-09-16--1633.patch against trunk revision 815628. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/92/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Dick!

        Show
        Chris Douglas added a comment - I committed this. Thanks, Dick!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #81 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/81/)
        . Use built-in interpolation classes for making up task
        runtimes in Rumen. Contributed by Dick King

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #81 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/81/ ) . Use built-in interpolation classes for making up task runtimes in Rumen. Contributed by Dick King
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #117 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/117/)
        . Use built-in interpolation classes for making up task
        runtimes in Rumen. Contributed by Dick King

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #117 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/117/ ) . Use built-in interpolation classes for making up task runtimes in Rumen. Contributed by Dick King

          People

          • Assignee:
            Dick King
            Reporter:
            Dick King
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development