Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2028

streaming should support MultiFileInputFormat

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: contrib/streaming
    • Labels:
      None

      Description

      There should be a way to call MultiFileInputFormat from streaming without having to write Java code...

        Activity

        Allen Wittenauer created issue -
        Hide
        Allen Wittenauer added a comment -

        Actually, what should probably happen is that MultiFileWordCount's "MyInputFormat" and "MultiLineRecordRecord" should get promoted out of examples and officially into the mapred(uce) APIs.

        The following appears to implement exactly what us streaming users want/need:

        $HADOOP_HOME/bin/hadoop \
        jar \
        `ls $HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar` \
        libjars `ls $HADOOP_HOME/hadoop*-examples.jar` \
        -inputformat org.apache.hadoop.examples.MultiFileWordCount\$MyInputFormat \
        -inputreader org.apache.hadoop.examples.MultiFileWordCount\$MultiFileLineRecordReader \
        ....

        Show
        Allen Wittenauer added a comment - Actually, what should probably happen is that MultiFileWordCount's "MyInputFormat" and "MultiLineRecordRecord" should get promoted out of examples and officially into the mapred(uce) APIs. The following appears to implement exactly what us streaming users want/need: $HADOOP_HOME/bin/hadoop \ jar \ `ls $HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar` \ libjars `ls $HADOOP_HOME/hadoop *-examples.jar` \ -inputformat org.apache.hadoop.examples.MultiFileWordCount\$MyInputFormat \ -inputreader org.apache.hadoop.examples.MultiFileWordCount\$MultiFileLineRecordReader \ ....
        Nigel Daley made changes -
        Field Original Value New Value
        Fix Version/s 0.22.0 [ 12314184 ]
        Fix Version/s 0.21.1 [ 12315272 ]
        Hide
        Allen Wittenauer added a comment -

        core devs don't use streaming so this won't get fixed.

        Show
        Allen Wittenauer added a comment - core devs don't use streaming so this won't get fixed.
        Allen Wittenauer made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        434d 18h 50m 1 Allen Wittenauer 02/Nov/11 17:39

          People

          • Assignee:
            Unassigned
            Reporter:
            Allen Wittenauer
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development