Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-619

Unify Map-Reduce and Streaming to take the same globbed input specification

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.1
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      Right now streaming input is specified very differently from other map-reduce input. It would be good if these two apps could take much more similar input specs.

      In particular -input in streaming expects a file or glob pattern while MR takes a directory. It would be cool if both could take a glob patern of files and if both took a directory by default (with some patern excluded to allow logs, metadata and other framework output to be safely stored).

      We want to be sure that MR input is backward compatible over this change. I propose that a single file should be accepted as an input or a single directory. Globs should only match directories if the paterns is '/' terminated, to avoid massive inputs specified by mistake.

      Thoughts?

        Attachments

        1. Hadoop-619_1.patch
          8 kB
          Sanjay Dahiya
        2. Hadoop-619_1.patch
          9 kB
          Sanjay Dahiya
        3. Hadoop-619_2.patch
          9 kB
          Sanjay Dahiya
        4. Hadoop-619_2.patch
          8 kB
          Sanjay Dahiya
        5. Hadoop-619_3.patch
          10 kB
          Sanjay Dahiya
        6. Hadoop-619_4.patch
          10 kB
          Doug Cutting
        7. Hadoop-619.patch
          12 kB
          Sanjay Dahiya
        8. Hadoop-619.patch
          12 kB
          Sanjay Dahiya
        9. Hadoop-619.patch
          11 kB
          Sanjay Dahiya

          Issue Links

            Activity

              People

              • Assignee:
                sanjay.dahiya Sanjay Dahiya
                Reporter:
                eric14 eric baldeschwieler
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: