Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-271 Change examples code to use new mapreduce api.
  3. MAPREDUCE-358

Change org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new mapreduce api.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Modifies AggregateWordCount and AggregateWordHistogram examples to use the new Map/Reduce API

      Description

      Change org.apache.hadoop.examples.AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new mapreduce api.

      1. patch-358.txt
        121 kB
        Amareshwari Sriramadasu
      2. patch-358-1.txt
        120 kB
        Amareshwari Sriramadasu
      3. patch-5689.txt
        109 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Amareshwari Sriramadasu created issue -
          Amareshwari Sriramadasu made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-5938 [ HADOOP-5938 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch modifies org.apache.hadoop.mapred.lib.aggregate to use new api. The new package is org.apache.hadoop.mapreduce.lib.aggregate. The old package is deprecated.
          Changes org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new api.

          Show
          Amareshwari Sriramadasu added a comment - Patch modifies org.apache.hadoop.mapred.lib.aggregate to use new api. The new package is org.apache.hadoop.mapreduce.lib.aggregate. The old package is deprecated. Changes org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new api.
          Amareshwari Sriramadasu made changes -
          Attachment patch-5689.txt [ 12410759 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result :

               [exec]
               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 8 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
          

          Tests in org.apache.hadoop.mapred.lib.aggregate and org.apache.hadoop.mapreduce.lib.aggregate passed on my machine.
          Ran the examples AggregateWordCount and AggregateWordHistogram on cluster and verified.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result : [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] Tests in org.apache.hadoop.mapred.lib.aggregate and org.apache.hadoop.mapreduce.lib.aggregate passed on my machine. Ran the examples AggregateWordCount and AggregateWordHistogram on cluster and verified.
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5689 MAPREDUCE-358
          Component/s mapred [ 12310690 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Hide
          Jothi Padmanabhan added a comment -

          I think we should try avoid code duplication wherever possible. For example, some static methods in the old api should just call the equivalent MapReduce.static method instead of duplicating code.
          So, the code for most of ValueAggregator*.java should be re-factored to avoid this duplication.
          Minor – UniqValueCount, the default constructor can be left out.

          Show
          Jothi Padmanabhan added a comment - I think we should try avoid code duplication wherever possible. For example, some static methods in the old api should just call the equivalent MapReduce.static method instead of duplicating code. So, the code for most of ValueAggregator*.java should be re-factored to avoid this duplication. Minor – UniqValueCount, the default constructor can be left out.
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          Removed duplicate code in static methods wherever possible.

          Show
          Amareshwari Sriramadasu added a comment - Removed duplicate code in static methods wherever possible.
          Amareshwari Sriramadasu made changes -
          Attachment patch-358.txt [ 12411493 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result:

               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 14 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
          

          run-test-mapred passed all tests on my machine.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result: [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 14 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] run-test-mapred passed all tests on my machine.
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Hadoop Flags [Incompatible change]
          Hide
          Jothi Padmanabhan added a comment -

          Changes look fine.
          One minor nit – Could you restore the java docs for the ValueAggregator* classes. Since they are public classes, I think the java doc should be present

          Show
          Jothi Padmanabhan added a comment - Changes look fine. One minor nit – Could you restore the java docs for the ValueAggregator* classes. Since they are public classes, I think the java doc should be present
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch restores javadoc for all the deprecated classes.

          test-patch and ant test passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - Patch restores javadoc for all the deprecated classes. test-patch and ant test passed on my machine.
          Amareshwari Sriramadasu made changes -
          Attachment patch-358-1.txt [ 12411621 ]
          Hide
          Jothi Padmanabhan added a comment -

          +1. Code changes look fine.

          Show
          Jothi Padmanabhan added a comment - +1. Code changes look fine.
          Hide
          Sharad Agarwal added a comment -

          I committed this. Thanks Amareshwari!

          Show
          Sharad Agarwal added a comment - I committed this. Thanks Amareshwari!
          Sharad Agarwal made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/ )
          Jothi Padmanabhan made changes -
          Release Note Modifies AggregateWordCount and AggregateWordHistogram examples to use the new Map/Reduce API
          Hide
          Meng Mao added a comment -

          Would it be possible to backport this into 0.20?
          Our current codebase is stuck running our aggregate-depending classes using deprecated API until the new mapreduce aggregate lib appears.

          Show
          Meng Mao added a comment - Would it be possible to backport this into 0.20? Our current codebase is stuck running our aggregate-depending classes using deprecated API until the new mapreduce aggregate lib appears.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          5d 21h 38m 1 Jothi Padmanabhan 22/Jun/09 05:33
          Open Open Patch Available Patch Available
          62d 1h 55m 2 Amareshwari Sriramadasu 23/Jun/09 05:52
          Patch Available Patch Available Resolved Resolved
          3d 1h 51m 1 Sharad Agarwal 26/Jun/09 07:44
          Resolved Resolved Closed Closed
          424d 14h 29m 1 Tom White 24/Aug/10 22:13

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development