Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1295

We need a job trace manipulator to build gridmix runs.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].

      It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.

      1. mapreduce-1297--2009-12-14.patch
        301 kB
        Dick King
      2. mapreduce-1295--2009-12-17.patch
        299 kB
        Dick King
      3. mapreduce-1295--2009-12-21.patch
        298 kB
        Dick King
      4. mapreduce-1295--2009-12-22.patch
        298 kB
        Dick King
      5. mapreduce-1295--2009-12-23.patch
        297 kB
        Dick King

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #200 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/200/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #200 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/200/ )
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Dick!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Dick!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428888/mapreduce-1295--2009-12-23.patch
        against trunk revision 893469.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428888/mapreduce-1295--2009-12-23.patch against trunk revision 893469. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/console This message is automatically generated.
        Hide
        Dick King added a comment -

        This addresses the issues raised earlier today.

        Show
        Dick King added a comment - This addresses the issues raised earlier today.
        Hide
        Dick King added a comment -

        This is a comment on the previous comment timestamped 23/Dec/09 02:59AM.

        • I will make all the DeskewedJobTraceReader constructors take a JobTraceReader . I see no reason why DeskewedJobTraceReader constructors shouldn't be public.
        • I will use FileSystem::exists .
        • I didn't notice that API when I wrote this code. My bad.
        • I don't want to require an empty directory because we use the output directory as the temp directory if none is supplied, and requiring an empty directory would make it impossible to do two runs into the same output directory.
        • I'll change the wording on some of the debug messages. I'll consider going to LOG4J .
        • I stand by my decision to use deletees . I don't like the deleteOnExit idiom because it makes it hard to use a tool as a callable component.
        Show
        Dick King added a comment - This is a comment on the previous comment timestamped 23/Dec/09 02:59AM. I will make all the DeskewedJobTraceReader constructors take a JobTraceReader . I see no reason why DeskewedJobTraceReader constructors shouldn't be public. I will use FileSystem::exists . I didn't notice that API when I wrote this code. My bad. I don't want to require an empty directory because we use the output directory as the temp directory if none is supplied, and requiring an empty directory would make it impossible to do two runs into the same output directory. I'll change the wording on some of the debug messages. I'll consider going to LOG4J . I stand by my decision to use deletees . I don't like the deleteOnExit idiom because it makes it hard to use a tool as a callable component.
        Hide
        Chris Douglas added a comment -

        The test failures are known (MAPREDUCE-1311, MAPREDUCE-1312).

        Only a few minor nits:

        • The two types of DeskewedJobTraceReader constructors could be combined by adding a private/protected cstr with a JobTraceReader formal
        • Should the System.err messages in DJTR::nextJob be debug messages?
        • The open/close idiom in run may be replaced by FileSystem::exists. Alternatively, require the user to provide a clean directory and use sequential segment numbering
        • The first person is a little disorienting in the debug log messages- which could use log4j loggers- but whatever you prefer
        • The deletees and similar accounting can be replace with FileSystem::deleteOnExit
        Show
        Chris Douglas added a comment - The test failures are known ( MAPREDUCE-1311 , MAPREDUCE-1312 ). Only a few minor nits: The two types of DeskewedJobTraceReader constructors could be combined by adding a private/protected cstr with a JobTraceReader formal Should the System.err messages in DJTR::nextJob be debug messages? The open/close idiom in run may be replaced by FileSystem::exists . Alternatively, require the user to provide a clean directory and use sequential segment numbering The first person is a little disorienting in the debug log messages- which could use log4j loggers- but whatever you prefer The deletees and similar accounting can be replace with FileSystem::deleteOnExit
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428754/mapreduce-1295--2009-12-22.patch
        against trunk revision 893055.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428754/mapreduce-1295--2009-12-22.patch against trunk revision 893055. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/console This message is automatically generated.
        Hide
        Dick King added a comment -

        Fprgot that RuntimeException implements Serializable and therefore any subclasses I define should have a serialVersionUID

        Show
        Dick King added a comment - Fprgot that RuntimeException implements Serializable and therefore any subclasses I define should have a serialVersionUID
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428686/mapreduce-1295--2009-12-21.patch
        against trunk revision 893055.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2331 javac compiler warnings (more than the trunk's current 2330 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428686/mapreduce-1295--2009-12-21.patch against trunk revision 893055. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2331 javac compiler warnings (more than the trunk's current 2330 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/console This message is automatically generated.
        Hide
        Dick King added a comment -

        This is the replacement for ...-17.patch

        Show
        Dick King added a comment - This is the replacement for ...-17.patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428351/mapreduce-1295--2009-12-17.patch
        against trunk revision 892117.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 2335 javac compiler warnings (more than the trunk's current 2331 warnings).

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428351/mapreduce-1295--2009-12-17.patch against trunk revision 892117. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2335 javac compiler warnings (more than the trunk's current 2331 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/console This message is automatically generated.
        Hide
        Dick King added a comment -

        This patch fixes an applicability issue.

        Show
        Dick King added a comment - This patch fixes an applicability issue.
        Hide
        Dick King added a comment -

        This patch applies on a direct download of Trunk, and replaces the previous patch

        Show
        Dick King added a comment - This patch applies on a direct download of Trunk, and replaces the previous patch
        Hide
        Chris Douglas added a comment -

        The patch doesn't apply; could you update it?

        Show
        Chris Douglas added a comment - The patch doesn't apply; could you update it?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427990/mapreduce-1297--2009-12-14.patch
        against trunk revision 890983.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/197/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427990/mapreduce-1297--2009-12-14.patch against trunk revision 890983. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/197/console This message is automatically generated.
        Hide
        Dick King added a comment -

        I misnamed the patch.

        I will need to fix it anyway; I noticed that it doesn't print out the needed skew buffer size. I will upload the new patch, with the correct name [and the then-correct date] after I see the hudson report and act on it if necessary.

        Show
        Dick King added a comment - I misnamed the patch. I will need to fix it anyway; I noticed that it doesn't print out the needed skew buffer size. I will upload the new patch, with the correct name [and the then-correct date] after I see the hudson report and act on it if necessary.
        Hide
        Dick King added a comment -

        This is the folder.

        The command line looks like this:

           Folder [ -output-duration duration ]           \
                  [ -input-cycle duration     ]           \
                  [ -concentration ratio      ]           \
                  [ -seed seed                ]           \
                  [ -temp temp-directory      ]           \
                  [ -debug                    ]           \
                  [ -skew-buffer-length n     ]           \
                  [ -allow-missorting         ]           \
               input-path output-path
        

        All paths and directories are general {{Path}}s.

        -output-duration is the length of the range of the submit times of the output trace.

        -input-cycle is the length of the input job submit time cycle. For example, if -input-cycle is 2 hours, then 3PM and 1PM is treated alike.

        -concentration is a double, a ratio of the density [number of jobs starting per hour] in the output over the input. This can be less than or greater than 1.0.

        -seed is a random number generator seed, used to create repeatable runs if desired. If no -seed is provided, we state what the -seed should be if you want to repeat this run.

        -temp is a temp directory, which must be able to hold about as much data as the input contains, compressed. Trace data compresses at about 17:1. This defaults to the directory of the output. The temporary files are erased whether the job succeeds or fails, unless -debug is coded.

        -debug induces the tool to produce a lot of debugging output, and causes the itnermediate files to be retained after a run.

        -skew-buffer-length describes the length of the skew buffer, which defaults to 0. The input to the folder should be sorted, and the job tracker log output is approximately sorted. However, there are occasional small glitches in the job tracker logs, jobs that come out a few places earlier than they should to be ordered by submit time. Code a -skew-buffer-length of i > 0 to allow as many as i jobs to arrive earlier than they're supposed to and be buffered until they can be released.

        -allow-missorting instructs the folder to just drop a job that arrives later than it should, if there's not enough room in the skew buffer for it. If -allow-missorting is not coded, we abend the run instead.

        If the run is successful, either because -allow-missorting is coded or -skew-buffer-length is big enough, the folder tells the user what the smallest -skew-buffer-length could have been for the run to succeed without omitting any jobs.

        Show
        Dick King added a comment - This is the folder. The command line looks like this: Folder [ -output-duration duration ] \ [ -input-cycle duration ] \ [ -concentration ratio ] \ [ -seed seed ] \ [ -temp temp-directory ] \ [ -debug ] \ [ -skew-buffer-length n ] \ [ -allow-missorting ] \ input-path output-path All paths and directories are general {{Path}}s. -output-duration is the length of the range of the submit times of the output trace. -input-cycle is the length of the input job submit time cycle. For example, if -input-cycle is 2 hours, then 3PM and 1PM is treated alike. -concentration is a double, a ratio of the density [number of jobs starting per hour] in the output over the input. This can be less than or greater than 1.0. -seed is a random number generator seed, used to create repeatable runs if desired. If no -seed is provided, we state what the -seed should be if you want to repeat this run. -temp is a temp directory, which must be able to hold about as much data as the input contains, compressed. Trace data compresses at about 17:1. This defaults to the directory of the output. The temporary files are erased whether the job succeeds or fails, unless -debug is coded. -debug induces the tool to produce a lot of debugging output, and causes the itnermediate files to be retained after a run. -skew-buffer-length describes the length of the skew buffer, which defaults to 0. The input to the folder should be sorted, and the job tracker log output is approximately sorted. However, there are occasional small glitches in the job tracker logs, jobs that come out a few places earlier than they should to be ordered by submit time. Code a -skew-buffer-length of i > 0 to allow as many as i jobs to arrive earlier than they're supposed to and be buffered until they can be released. -allow-missorting instructs the folder to just drop a job that arrives later than it should, if there's not enough room in the skew buffer for it. If -allow-missorting is not coded, we abend the run instead. If the run is successful, either because -allow-missorting is coded or -skew-buffer-length is big enough, the folder tells the user what the smallest -skew-buffer-length could have been for the run to succeed without omitting any jobs.

          People

          • Assignee:
            Dick King
            Reporter:
            Dick King
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development