Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1461

Feature to instruct rumen-folder utility to skip jobs worth of specific duration

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added a ''-starts-after' option to Rumen's Folder utility. The time duration specified after the '-starts-after' option is an offset with respect to the submit time of the first job in the input trace. Jobs in the input trace having a submit time (relative to the first job's submit time) lesser than the specified offset will be ignored.
      Show
      Added a ''-starts-after' option to Rumen's Folder utility. The time duration specified after the '-starts-after' option is an offset with respect to the submit time of the first job in the input trace. Jobs in the input trace having a submit time (relative to the first job's submit time) lesser than the specified offset will be ignored.
    • Tags:
      rumen, folder, skip-jobs

      Description

      JSON outputs of rumen on production logs can be huge in the order of multiple GB. Rumen's folder utility helps in getting a smaller snapshot of this JSON data.
      It would be helpful to have an option in rumen-folder, wherein user can specify a duration from which rumen-folder should start processing data.

      Related JIRA link: https://issues.apache.org/jira/browse/MAPREDUCE-1295

      1. mr-1461-trunk-with-testcases.patch
        8 kB
        Rajesh Balamohan
      2. MR-1461-trunk.patch
        6 kB
        Rajesh Balamohan
      3. mapreduce-1461--2010-03-04.patch
        2 kB
        Rajesh Balamohan
      4. mapreduce-1461--2010-02-05.patch
        2 kB
        Rajesh Balamohan

        Activity

        Hide
        Amar Kamat added a comment -

        I just committed this. Thanks Rajesh!

        Show
        Amar Kamat added a comment - I just committed this. Thanks Rajesh!
        Hide
        Amar Kamat added a comment -

        The failed testcases are not related to this patch. I will commit this now.

        Show
        Amar Kamat added a comment - The failed testcases are not related to this patch. I will commit this now.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12477476/mr-1461-trunk-with-testcases.patch
        against trunk revision 1096649.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.tools.TestHadoopArchives
        org.apache.hadoop.tools.TestHarFileSystem

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477476/mr-1461-trunk-with-testcases.patch against trunk revision 1096649. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/188//console This message is automatically generated.
        Hide
        Rajesh Balamohan added a comment -

        Attaching the patch with -ve testcase as well.

        Show
        Rajesh Balamohan added a comment - Attaching the patch with -ve testcase as well.
        Hide
        Amar Kamat added a comment -

        Rajesh, the patch looks good. Can you also add a negative testcase for skipped jobs?

        Show
        Amar Kamat added a comment - Rajesh, the patch looks good. Can you also add a negative testcase for skipped jobs?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12475669/MR-1461-trunk.patch
        against trunk revision 1094093.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475669/MR-1461-trunk.patch against trunk revision 1094093. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/171//console This message is automatically generated.
        Hide
        Rajesh Balamohan added a comment -

        Regenerated the patch for latest apache trunk codebase

        Show
        Rajesh Balamohan added a comment - Regenerated the patch for latest apache trunk codebase
        Hide
        Rajesh Balamohan added a comment -

        I took the trunk version and generated the patch. Please refer the attached file.

        Show
        Rajesh Balamohan added a comment - I took the trunk version and generated the patch. Please refer the attached file.
        Hide
        Chris Douglas added a comment -

        Please regenerate the patch relative to the root of the source tree

        Show
        Chris Douglas added a comment - Please regenerate the patch relative to the root of the source tree
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12434931/mapreduce-1461--2010-02-05.patch
        against trunk revision 909241.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/318/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12434931/mapreduce-1461--2010-02-05.patch against trunk revision 909241. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/318/console This message is automatically generated.
        Hide
        Hong Tang added a comment -

        Mark patch available for hudson to pick up.

        Show
        Hong Tang added a comment - Mark patch available for hudson to pick up.
        Hide
        Rajesh Balamohan added a comment -

        The attached patch implements this feature. User can specify the time duration to be skipped by specifying "-starts-after" commandline argument.

        Show
        Rajesh Balamohan added a comment - The attached patch implements this feature. User can specify the time duration to be skipped by specifying "-starts-after" commandline argument.

          People

          • Assignee:
            Rajesh Balamohan
            Reporter:
            Rajesh Balamohan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development