Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2028

streaming should support MultiFileInputFormat

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: contrib/streaming
    • Labels:
      None

      Description

      There should be a way to call MultiFileInputFormat from streaming without having to write Java code...

        Activity

        Hide
        Allen Wittenauer added a comment -

        core devs don't use streaming so this won't get fixed.

        Show
        Allen Wittenauer added a comment - core devs don't use streaming so this won't get fixed.
        Hide
        Allen Wittenauer added a comment -

        Actually, what should probably happen is that MultiFileWordCount's "MyInputFormat" and "MultiLineRecordRecord" should get promoted out of examples and officially into the mapred(uce) APIs.

        The following appears to implement exactly what us streaming users want/need:

        $HADOOP_HOME/bin/hadoop \
        jar \
        `ls $HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar` \
        libjars `ls $HADOOP_HOME/hadoop*-examples.jar` \
        -inputformat org.apache.hadoop.examples.MultiFileWordCount\$MyInputFormat \
        -inputreader org.apache.hadoop.examples.MultiFileWordCount\$MultiFileLineRecordReader \
        ....

        Show
        Allen Wittenauer added a comment - Actually, what should probably happen is that MultiFileWordCount's "MyInputFormat" and "MultiLineRecordRecord" should get promoted out of examples and officially into the mapred(uce) APIs. The following appears to implement exactly what us streaming users want/need: $HADOOP_HOME/bin/hadoop \ jar \ `ls $HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar` \ libjars `ls $HADOOP_HOME/hadoop *-examples.jar` \ -inputformat org.apache.hadoop.examples.MultiFileWordCount\$MyInputFormat \ -inputreader org.apache.hadoop.examples.MultiFileWordCount\$MultiFileLineRecordReader \ ....

          People

          • Assignee:
            Unassigned
            Reporter:
            Allen Wittenauer
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development