Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-813

Streaming Doc and M/R-Tutorial Doc - updates

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA addresses issues in the Streaming doc that also require a cross-link to and update in the M/R Tutorial doc. All changes approved by the reviewing engineer.

      Streaming issues:

      1. During the execution of a streaming job, the names of the "mapred" parameters are transformed. The dots ( . ) become underscores ( _ ).

      Docs affected: streaming and m/r tutorial (new sub-sections added under Task Execution & Environment section)

      2. For -files and -archives options, Hadoop now creates symlink with same name as file (user-defined symlinks, #mysymlink, currently not supported)

      Docs affected:streaming

      3. Streaming supports streaming command options and generic command options. Generic options must be placed before streaming options, otherwise command fails.

      Docs affected: streaming (reorganized the streaming doc to make distinctions between 2 sets of command options more clear)

      1. ASF.LICENSE.NOT.GRANTED--patch-813-0.20.txt
        46 kB
        Amareshwari Sriramadasu
      2. patch-813-ydist.txt
        47 kB
        Amareshwari Sriramadasu
      3. MAPREDUCE-813-3.patch
        67 kB
        Devaraj Das
      4. MAPREDUCE-813-2.patch
        46 kB
        Corinne Chandel
      5. MAPREDUCE-813.patch
        44 kB
        Corinne Chandel
      6. streaming.pdf
        62 kB
        Corinne Chandel
      7. mapred_tutorial.pdf
        159 kB
        Corinne Chandel

        Issue Links

          Activity

          Hide
          Corinne Chandel added a comment -

          1. mapred_tutorial.pdf - shows doc changes

          2. streaming.pdf - shows doc changes

          3. MAPREDUCE-813.patch - patch file

          Show
          Corinne Chandel added a comment - 1. mapred_tutorial.pdf - shows doc changes 2. streaming.pdf - shows doc changes 3. MAPREDUCE-813 .patch - patch file
          Hide
          Corinne Chandel added a comment -

          Apply this patch to the hadoop-m/r branch: https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk

          Note: No new test code required; changes to documentation only.

          Show
          Corinne Chandel added a comment - Apply this patch to the hadoop-m/r branch: https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk Note: No new test code required; changes to documentation only.
          Hide
          Amareshwari Sriramadasu added a comment -

          Some minor comments:
          1. mapred_tutorial.xml has some tabs instead of spaces.
          2.

           <p>
          
          -Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:
          
          +Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run map/reduce jobs with any executable or 
          
          +script as the mapper and/or the reducer. For example:
          
           </p>
          
          

          The documentation change here is the change from Map/Reduce to map/reduce. But HADOOP-3691 made sure in all the places, it should be Map/Reduce. I see the same change in many places. Was this intentional?

          Show
          Amareshwari Sriramadasu added a comment - Some minor comments: 1. mapred_tutorial.xml has some tabs instead of spaces. 2. <p> -Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example: +Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run map/reduce jobs with any executable or +script as the mapper and/or the reducer. For example: </p> The documentation change here is the change from Map/Reduce to map/reduce . But HADOOP-3691 made sure in all the places, it should be Map/Reduce. I see the same change in many places. Was this intentional?
          Hide
          Corinne Chandel added a comment -

          Patch file #2.

          (1) replaced map/reduce with MAP/REDUCE (m/r tutorial doc, streaming doc)

          (2) removed tabs (m/r tutorial doc)

          Show
          Corinne Chandel added a comment - Patch file #2. (1) replaced map/reduce with MAP/REDUCE (m/r tutorial doc, streaming doc) (2) removed tabs (m/r tutorial doc)
          Hide
          Amareshwari Sriramadasu added a comment -

          Document changes look good. But some minor comments on indentation :
          1. mapred_tutorial.xml has changes like

          -  
          +
          
          and 
          -              <code>package org.myorg;</code>
          +                <code>package org.myorg;</code>
          
          

          etc.

          This are not necessary.

          {code
          + <section>
          + <title>Configured Parameters</title>

          More spaces for title are not necessary. (Not there for other sections).
          
          2. In steaming.xml,
          

          -Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:

          +Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or

          +script as the mapper and/or the reducer. For example:

          
          

          Change here is unnecessary. If you want, you can make the lines fit into 80 character lines.
          The same applies to similar changes in the doc.

          Show
          Amareshwari Sriramadasu added a comment - Document changes look good. But some minor comments on indentation : 1. mapred_tutorial.xml has changes like - + and - <code> package org.myorg;</code> + <code> package org.myorg;</code> etc. This are not necessary. {code + <section> + <title>Configured Parameters</title> More spaces for title are not necessary. (Not there for other sections). 2. In steaming.xml, -Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example: +Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or +script as the mapper and/or the reducer. For example: Change here is unnecessary. If you want, you can make the lines fit into 80 character lines. The same applies to similar changes in the doc.
          Hide
          Corinne Chandel added a comment -

          I'm not cleaning up the indentation on the XML files.
          If there are not other content changes, then we can go with MAPREDUCE-813-2.patch.

          Thanks/C

          Show
          Corinne Chandel added a comment - I'm not cleaning up the indentation on the XML files. If there are not other content changes, then we can go with MAPREDUCE-813 -2.patch. Thanks/C
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12415063/MAPREDUCE-813-2.patch
          against trunk revision 800329.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/438/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415063/MAPREDUCE-813-2.patch against trunk revision 800329. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/438/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          For some reason, the 813-2.patch doesn't apply cleanly on trunk. Attaching a patch that applies cleanly on trunk

          Show
          Devaraj Das added a comment - For some reason, the 813-2.patch doesn't apply cleanly on trunk. Attaching a patch that applies cleanly on trunk
          Hide
          Amareshwari Sriramadasu added a comment -

          I ran forrest on my machine with patch. It is Successful.

          Show
          Amareshwari Sriramadasu added a comment - I ran forrest on my machine with patch. It is Successful.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Corinne!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Corinne!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/ )
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for Yahoo! distribution.
          Ran ant docs and it was successful.

          Show
          Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution. Ran ant docs and it was successful.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for branch 0.20.

          Show
          Amareshwari Sriramadasu added a comment - Patch for branch 0.20.
          Hide
          Amareshwari Sriramadasu added a comment -

          I ran ant docs with the patch for 0.20 : https://issues.apache.org/jira/secure/attachment/12441581/patch-813-0.20.txt. It ran successfully. Can somebody please commit the patch to branch 0.20?

          Show
          Amareshwari Sriramadasu added a comment - I ran ant docs with the patch for 0.20 : https://issues.apache.org/jira/secure/attachment/12441581/patch-813-0.20.txt . It ran successfully. Can somebody please commit the patch to branch 0.20?

            People

            • Assignee:
              Unassigned
              Reporter:
              Corinne Chandel
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development