Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Rumen could expose a cleaner interface to simplify the integration with other tools.

      1. mapreduce-966-20090910-4.patch
        124 kB
        Hong Tang
      2. mapreduce-966-20090910-3.patch
        124 kB
        Hong Tang

        Issue Links

          Activity

          Hide
          Hong Tang added a comment -

          Proposed changes:

          • Isolate the tools dependent on Rumen to three simple interfaces:
            1. JobStory (describing a MapReduce job).
            2. ClusterStory (describing the cluster setup and topology etc)
            3. JobStoryProducer that produces a sequence of jobs.

          Accordingly, ZombieJob adapts a LoggedJob to JobStory, and ZombieCluster adapts LoggedNetworkTopology to ClusterStory (indirectly through AbstractClusterStory). Finally ZombieJobProducer reads rumen traces and produces a sequence of JobStory instances.

          • Encapsulate the logic of JSON parsing within Rumen and remove the Parser class. Two reader classes are added to parse json encoded LoggedJob and LoggedNetworkTopology (JobTraceReader and ClusterTopologyReader). No throw of Json-specific exceptions from the interface.
          • Better sanity check in ZombieJob, and fill in made-up data if the source data are missing or invalid.
          Show
          Hong Tang added a comment - Proposed changes: Isolate the tools dependent on Rumen to three simple interfaces: JobStory (describing a MapReduce job). ClusterStory (describing the cluster setup and topology etc) JobStoryProducer that produces a sequence of jobs. Accordingly, ZombieJob adapts a LoggedJob to JobStory, and ZombieCluster adapts LoggedNetworkTopology to ClusterStory (indirectly through AbstractClusterStory). Finally ZombieJobProducer reads rumen traces and produces a sequence of JobStory instances. Encapsulate the logic of JSON parsing within Rumen and remove the Parser class. Two reader classes are added to parse json encoded LoggedJob and LoggedNetworkTopology (JobTraceReader and ClusterTopologyReader). No throw of Json-specific exceptions from the interface. Better sanity check in ZombieJob, and fill in made-up data if the source data are missing or invalid.
          Hide
          Hong Tang added a comment -

          The attached patch implements the changes proposed above. During the process, we also fixed a few minor issues:

          • Changed the usage of java io API (File etc) to hadoop Path, FileSystem, Configuration.
          • Upgraded the tests to junit 4.
          • Use the newly added JsonObjectMapperParser to replace customized json parsing in TestRumenJobTraces.
          • Replaced the usage of Vector to List in unit tests.
          • Fixed an NPE bug in HadoopLogAnalyzer.
          • Added an API getOutcome() in JobStory.
          • Fixed a bug in ParsedHost where it fails to parse rack names contain non-digital characters.
          • Fixed a bug where ZombieJob.getTaskInfo() returns uninitialized TaskInfo objects.
          • Adapt the usage of newly added ClusterStory in ZombieJob.
          Show
          Hong Tang added a comment - The attached patch implements the changes proposed above. During the process, we also fixed a few minor issues: Changed the usage of java io API (File etc) to hadoop Path, FileSystem, Configuration. Upgraded the tests to junit 4. Use the newly added JsonObjectMapperParser to replace customized json parsing in TestRumenJobTraces. Replaced the usage of Vector to List in unit tests. Fixed an NPE bug in HadoopLogAnalyzer. Added an API getOutcome() in JobStory. Fixed a bug in ParsedHost where it fails to parse rack names contain non-digital characters. Fixed a bug where ZombieJob.getTaskInfo() returns uninitialized TaskInfo objects. Adapt the usage of newly added ClusterStory in ZombieJob.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419252/mapreduce-966-20090910-3.patch
          against trunk revision 813585.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419252/mapreduce-966-20090910-3.patch against trunk revision 813585. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/console This message is automatically generated.
          Hide
          Hong Tang added a comment -

          Patch addresses the findbugs warnings.

          Show
          Hong Tang added a comment - Patch addresses the findbugs warnings.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419278/mapreduce-966-20090910-4.patch
          against trunk revision 813660.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419278/mapreduce-966-20090910-4.patch against trunk revision 813660. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Hong!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Hong!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/32/)
          . Modify Rumen to clean up interfaces and simplify integration
          with other tools. Contributed by Hong Tang

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/32/ ) . Modify Rumen to clean up interfaces and simplify integration with other tools. Contributed by Hong Tang

            People

            • Assignee:
              Hong Tang
              Reporter:
              Hong Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development