Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Rumen could expose a cleaner interface to simplify the integration with other tools.

      1. mapreduce-966-20090910-3.patch
        124 kB
        Hong Tang
      2. mapreduce-966-20090910-4.patch
        124 kB
        Hong Tang

        Issue Links

          Activity

          Hong Tang created issue -
          Hide
          Hong Tang added a comment -

          Proposed changes:

          • Isolate the tools dependent on Rumen to three simple interfaces:
            1. JobStory (describing a MapReduce job).
            2. ClusterStory (describing the cluster setup and topology etc)
            3. JobStoryProducer that produces a sequence of jobs.

          Accordingly, ZombieJob adapts a LoggedJob to JobStory, and ZombieCluster adapts LoggedNetworkTopology to ClusterStory (indirectly through AbstractClusterStory). Finally ZombieJobProducer reads rumen traces and produces a sequence of JobStory instances.

          • Encapsulate the logic of JSON parsing within Rumen and remove the Parser class. Two reader classes are added to parse json encoded LoggedJob and LoggedNetworkTopology (JobTraceReader and ClusterTopologyReader). No throw of Json-specific exceptions from the interface.
          • Better sanity check in ZombieJob, and fill in made-up data if the source data are missing or invalid.
          Show
          Hong Tang added a comment - Proposed changes: Isolate the tools dependent on Rumen to three simple interfaces: JobStory (describing a MapReduce job). ClusterStory (describing the cluster setup and topology etc) JobStoryProducer that produces a sequence of jobs. Accordingly, ZombieJob adapts a LoggedJob to JobStory, and ZombieCluster adapts LoggedNetworkTopology to ClusterStory (indirectly through AbstractClusterStory). Finally ZombieJobProducer reads rumen traces and produces a sequence of JobStory instances. Encapsulate the logic of JSON parsing within Rumen and remove the Parser class. Two reader classes are added to parse json encoded LoggedJob and LoggedNetworkTopology (JobTraceReader and ClusterTopologyReader). No throw of Json-specific exceptions from the interface. Better sanity check in ZombieJob, and fill in made-up data if the source data are missing or invalid.
          Hong Tang made changes -
          Field Original Value New Value
          Attachment mapreduce-966-20090910-3.patch [ 12419252 ]
          Hide
          Hong Tang added a comment -

          The attached patch implements the changes proposed above. During the process, we also fixed a few minor issues:

          • Changed the usage of java io API (File etc) to hadoop Path, FileSystem, Configuration.
          • Upgraded the tests to junit 4.
          • Use the newly added JsonObjectMapperParser to replace customized json parsing in TestRumenJobTraces.
          • Replaced the usage of Vector to List in unit tests.
          • Fixed an NPE bug in HadoopLogAnalyzer.
          • Added an API getOutcome() in JobStory.
          • Fixed a bug in ParsedHost where it fails to parse rack names contain non-digital characters.
          • Fixed a bug where ZombieJob.getTaskInfo() returns uninitialized TaskInfo objects.
          • Adapt the usage of newly added ClusterStory in ZombieJob.
          Show
          Hong Tang added a comment - The attached patch implements the changes proposed above. During the process, we also fixed a few minor issues: Changed the usage of java io API (File etc) to hadoop Path, FileSystem, Configuration. Upgraded the tests to junit 4. Use the newly added JsonObjectMapperParser to replace customized json parsing in TestRumenJobTraces. Replaced the usage of Vector to List in unit tests. Fixed an NPE bug in HadoopLogAnalyzer. Added an API getOutcome() in JobStory. Fixed a bug in ParsedHost where it fails to parse rack names contain non-digital characters. Fixed a bug where ZombieJob.getTaskInfo() returns uninitialized TaskInfo objects. Adapt the usage of newly added ClusterStory in ZombieJob.
          Hong Tang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419252/mapreduce-966-20090910-3.patch
          against trunk revision 813585.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419252/mapreduce-966-20090910-3.patch against trunk revision 813585. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/console This message is automatically generated.
          Hide
          Hong Tang added a comment -

          Patch addresses the findbugs warnings.

          Show
          Hong Tang added a comment - Patch addresses the findbugs warnings.
          Hong Tang made changes -
          Attachment mapreduce-966-20090910-4.patch [ 12419278 ]
          Hong Tang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hong Tang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Link This issue blocks MAPREDUCE-776 [ MAPREDUCE-776 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419278/mapreduce-966-20090910-4.patch
          against trunk revision 813660.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419278/mapreduce-966-20090910-4.patch against trunk revision 813660. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Hong!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Hong!
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/32/)
          . Modify Rumen to clean up interfaces and simplify integration
          with other tools. Contributed by Hong Tang

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/32/ ) . Modify Rumen to clean up interfaces and simplify integration with other tools. Contributed by Hong Tang
          Amar Kamat made changes -
          Component/s tools/rumen [ 12313617 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          5h 18m 1 Hong Tang 11/Sep/09 06:57
          Open Open Patch Available Patch Available
          1d 3h 45m 2 Hong Tang 11/Sep/09 06:57
          Patch Available Patch Available Resolved Resolved
          1d 2h 33m 1 Chris Douglas 12/Sep/09 09:31
          Resolved Resolved Closed Closed
          346d 11h 45m 1 Tom White 24/Aug/10 21:17

            People

            • Assignee:
              Hong Tang
              Reporter:
              Hong Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development