Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-271 Change examples code to use new mapreduce api.
  3. MAPREDUCE-358

Change org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new mapreduce api.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Modifies AggregateWordCount and AggregateWordHistogram examples to use the new Map/Reduce API

      Description

      Change org.apache.hadoop.examples.AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new mapreduce api.

      1. patch-5689.txt
        109 kB
        Amareshwari Sriramadasu
      2. patch-358-1.txt
        120 kB
        Amareshwari Sriramadasu
      3. patch-358.txt
        121 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Amareshwari Sriramadasu added a comment -

          Patch modifies org.apache.hadoop.mapred.lib.aggregate to use new api. The new package is org.apache.hadoop.mapreduce.lib.aggregate. The old package is deprecated.
          Changes org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new api.

          Show
          Amareshwari Sriramadasu added a comment - Patch modifies org.apache.hadoop.mapred.lib.aggregate to use new api. The new package is org.apache.hadoop.mapreduce.lib.aggregate. The old package is deprecated. Changes org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new api.
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result :

               [exec]
               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 8 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
          

          Tests in org.apache.hadoop.mapred.lib.aggregate and org.apache.hadoop.mapreduce.lib.aggregate passed on my machine.
          Ran the examples AggregateWordCount and AggregateWordHistogram on cluster and verified.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result : [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] Tests in org.apache.hadoop.mapred.lib.aggregate and org.apache.hadoop.mapreduce.lib.aggregate passed on my machine. Ran the examples AggregateWordCount and AggregateWordHistogram on cluster and verified.
          Hide
          Jothi Padmanabhan added a comment -

          I think we should try avoid code duplication wherever possible. For example, some static methods in the old api should just call the equivalent MapReduce.static method instead of duplicating code.
          So, the code for most of ValueAggregator*.java should be re-factored to avoid this duplication.
          Minor – UniqValueCount, the default constructor can be left out.

          Show
          Jothi Padmanabhan added a comment - I think we should try avoid code duplication wherever possible. For example, some static methods in the old api should just call the equivalent MapReduce.static method instead of duplicating code. So, the code for most of ValueAggregator*.java should be re-factored to avoid this duplication. Minor – UniqValueCount, the default constructor can be left out.
          Hide
          Amareshwari Sriramadasu added a comment -

          Removed duplicate code in static methods wherever possible.

          Show
          Amareshwari Sriramadasu added a comment - Removed duplicate code in static methods wherever possible.
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result:

               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 14 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
          

          run-test-mapred passed all tests on my machine.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result: [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 14 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] run-test-mapred passed all tests on my machine.
          Hide
          Jothi Padmanabhan added a comment -

          Changes look fine.
          One minor nit – Could you restore the java docs for the ValueAggregator* classes. Since they are public classes, I think the java doc should be present

          Show
          Jothi Padmanabhan added a comment - Changes look fine. One minor nit – Could you restore the java docs for the ValueAggregator* classes. Since they are public classes, I think the java doc should be present
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch restores javadoc for all the deprecated classes.

          test-patch and ant test passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - Patch restores javadoc for all the deprecated classes. test-patch and ant test passed on my machine.
          Hide
          Jothi Padmanabhan added a comment -

          +1. Code changes look fine.

          Show
          Jothi Padmanabhan added a comment - +1. Code changes look fine.
          Hide
          Sharad Agarwal added a comment -

          I committed this. Thanks Amareshwari!

          Show
          Sharad Agarwal added a comment - I committed this. Thanks Amareshwari!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/ )
          Hide
          Meng Mao added a comment -

          Would it be possible to backport this into 0.20?
          Our current codebase is stuck running our aggregate-depending classes using deprecated API until the new mapreduce aggregate lib appears.

          Show
          Meng Mao added a comment - Would it be possible to backport this into 0.20? Our current codebase is stuck running our aggregate-depending classes using deprecated API until the new mapreduce aggregate lib appears.

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development