Hadoop Common
  1. Hadoop Common
  2. HADOOP-1722

Make streaming to handle non-utf8 byte array

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.2, 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Streaming allows binary (or other non-UTF8) streams.

      Description

      Right now, the streaming framework expects the output sof the steam process (mapper or reducer) are line
      oriented UTF-8 text. This limit makes it impossible to use those programs whose outputs may be non-UTF-8
      (international encoding, or maybe even binary data). Streaming can overcome this limit by introducing a simple
      encoding protocol. For example, it can allow the mapper/reducer to hexencode its keys/values,
      the framework decodes them in the Java side.
      This way, as long as the mapper/reducer executables follow this encoding protocol,
      they can output arabitary bytearray and the streaming framework can handle them.

      1. HADOOP-1722-v0.20.1.patch
        153 kB
        Matthias Lehmann
      2. HADOOP-1722-branch-0.19.patch
        152 kB
        Klaas Bosteels
      3. HADOOP-1722-branch-0.18.patch
        152 kB
        Klaas Bosteels
      4. HADOOP-1722-v6.patch
        153 kB
        Klaas Bosteels
      5. HADOOP-1722-v5.patch
        153 kB
        Klaas Bosteels
      6. HADOOP-1722-v4.patch
        121 kB
        Klaas Bosteels
      7. HADOOP-1722-v4.patch
        119 kB
        Klaas Bosteels
      8. HADOOP-1722-v3.patch
        119 kB
        Klaas Bosteels
      9. HADOOP-1722-v2.patch
        119 kB
        Klaas Bosteels
      10. HADOOP-1722.patch
        114 kB
        Klaas Bosteels

        Issue Links

          Activity

            People

            • Assignee:
              Klaas Bosteels
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development