Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1547

Provide examples for aggregate library

        Details

        • Type: Improvement
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.14.0
        • Component/s: None
        • Labels:
          None

          Description

          The abacus examples were not moved to aggregate when it was created. Provide an aggregate example which shows users how to use the aggregate library.

          As a part of this issue move TestAggregates to the org.apache.hadoop.mapred.lib.aggregate package.

          1. Text File
            patch_1547.txt
            27 kB
            Runping Qi

            Issue Links

            blocks

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-1302 Remove deprecated contrib/abacus code

            • Major - Major loss of function.
            • Closed
            depends upon

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-1 Hadoop mapreduce should always ship the jar file(s) specified by the user

            • Major - Major loss of function.
            • Resolved
            is related to

            New Feature - A new feature of the product, which has yet to be developed. HADOOP-908 Hadoop Abacus, a package for performing simple counting/aggregation

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              runping Runping Qi added a comment -

              A proper example will not work before HADOOP-1521 is fixed.

              Show
              runping Runping Qi added a comment - A proper example will not work before HADOOP-1521 is fixed.
              Hide
              Permalink
              runping Runping Qi added a comment -

              This patch addresses the following issues:

              1. Move the TestAggregate to mapred.lib.aggregate package.

              2. Provide an example in the examples dir.
              The example is to use Aggregate to count the word occurrences in the input files.

              3. Replace work Abacus in the comments in the source files with Aggregate

              4. Modify the ValueAggregatorJob so that it allows the user to specify a user jar file
              in the spec file under attribute anme "user.jar.file".
              This makes Aggregate package work without depending on HADOOP-1521

              Show
              runping Runping Qi added a comment - This patch addresses the following issues: 1. Move the TestAggregate to mapred.lib.aggregate package. 2. Provide an example in the examples dir. The example is to use Aggregate to count the word occurrences in the input files. 3. Replace work Abacus in the comments in the source files with Aggregate 4. Modify the ValueAggregatorJob so that it allows the user to specify a user jar file in the spec file under attribute anme "user.jar.file". This makes Aggregate package work without depending on HADOOP-1521
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              +1

              http://issues.apache.org/jira/secure/attachment/12361439/patch_1547.txt applied and successfully tested against trunk revision r554144.

              Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/379/testReport/
              Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/379/console

              Show
              hadoopqa Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12361439/patch_1547.txt applied and successfully tested against trunk revision r554144. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/379/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/379/console
              Hide
              Permalink
              runping Runping Qi added a comment -

              I hope this patch can get in the next release.
              Let me know if there are any concerns for committing it.

              Show
              runping Runping Qi added a comment - I hope this patch can get in the next release. Let me know if there are any concerns for committing it.
              Hide
              Permalink
              owen.omalley Owen O'Malley added a comment -

              Runping, can you please reformat the patch to stay within 80 characters?

              Show
              owen.omalley Owen O'Malley added a comment - Runping, can you please reformat the patch to stay within 80 characters?
              Hide
              Permalink
              tomwhite Tom White added a comment -

              Runping, please also update the usage line in the javadoc of AggregateWordCount to include 

              1 textinputformat

              to specify the number of reducers and the type of input. When I tried omitting these I got an exception.

              Show
              tomwhite Tom White added a comment - Runping, please also update the usage line in the javadoc of AggregateWordCount to include 1 textinputformat to specify the number of reducers and the type of input. When I tried omitting these I got an exception.
              Hide
              Permalink
              runping Runping Qi added a comment -


              A new patch addressing Tom and Owen's comments attached.

              Show
              runping Runping Qi added a comment - A new patch addressing Tom and Owen's comments attached.
              Hide
              Permalink
              tomwhite Tom White added a comment -

              I've just committed this. Thanks Runping!

              I made a further minor change to the usage javadoc.

              Show
              tomwhite Tom White added a comment - I've just committed this. Thanks Runping! I made a further minor change to the usage javadoc.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1, could not apply patch.

              The patch command could not apply the latest attachment http://issues.apache.org/jira/secure/attachment/12361701/patch_1547.txt as a patch to trunk revision r555770.

              Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/401/console

              Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

              Show
              hadoopqa Hadoop QA added a comment - -1, could not apply patch. The patch command could not apply the latest attachment http://issues.apache.org/jira/secure/attachment/12361701/patch_1547.txt as a patch to trunk revision r555770. Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/401/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Nightly #154 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/154/)

              Show
              hudson Hudson added a comment - Integrated in Hadoop-Nightly #154 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/154/ )

                People

                • Assignee:
                  runping Runping Qi
                  Reporter:
                  tomwhite Tom White
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development