Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1722

Make streaming to handle non-utf8 byte array

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.2, 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Streaming allows binary (or other non-UTF8) streams.

      Description

      Right now, the streaming framework expects the output sof the steam process (mapper or reducer) are line
      oriented UTF-8 text. This limit makes it impossible to use those programs whose outputs may be non-UTF-8
      (international encoding, or maybe even binary data). Streaming can overcome this limit by introducing a simple
      encoding protocol. For example, it can allow the mapper/reducer to hexencode its keys/values,
      the framework decodes them in the Java side.
      This way, as long as the mapper/reducer executables follow this encoding protocol,
      they can output arabitary bytearray and the streaming framework can handle them.

        Attachments

        1. HADOOP-1722-v6.patch
          153 kB
          Klaas Bosteels
        2. HADOOP-1722-v5.patch
          153 kB
          Klaas Bosteels
        3. HADOOP-1722-v4.patch
          119 kB
          Klaas Bosteels
        4. HADOOP-1722-v4.patch
          121 kB
          Klaas Bosteels
        5. HADOOP-1722-v3.patch
          119 kB
          Klaas Bosteels
        6. HADOOP-1722-v2.patch
          119 kB
          Klaas Bosteels
        7. HADOOP-1722-v0.20.1.patch
          153 kB
          Matthias Lehmann
        8. HADOOP-1722-branch-0.19.patch
          152 kB
          Klaas Bosteels
        9. HADOOP-1722-branch-0.18.patch
          152 kB
          Klaas Bosteels
        10. HADOOP-1722.patch
          114 kB
          Klaas Bosteels

          Issue Links

            Activity

              People

              • Assignee:
                klbostee Klaas Bosteels
                Reporter:
                runping Runping Qi
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: