Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1722

Make streaming to handle non-utf8 byte array

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.0.2, 0.21.0
    • None
    • None
    • Reviewed
    • Streaming allows binary (or other non-UTF8) streams.

    Description

      Right now, the streaming framework expects the output sof the steam process (mapper or reducer) are line
      oriented UTF-8 text. This limit makes it impossible to use those programs whose outputs may be non-UTF-8
      (international encoding, or maybe even binary data). Streaming can overcome this limit by introducing a simple
      encoding protocol. For example, it can allow the mapper/reducer to hexencode its keys/values,
      the framework decodes them in the Java side.
      This way, as long as the mapper/reducer executables follow this encoding protocol,
      they can output arabitary bytearray and the streaming framework can handle them.

      Attachments

        1. HADOOP-1722.patch
          114 kB
          Klaas Bosteels
        2. HADOOP-1722-branch-0.18.patch
          152 kB
          Klaas Bosteels
        3. HADOOP-1722-branch-0.19.patch
          152 kB
          Klaas Bosteels
        4. HADOOP-1722-v0.20.1.patch
          153 kB
          Matthias Lehmann
        5. HADOOP-1722-v2.patch
          119 kB
          Klaas Bosteels
        6. HADOOP-1722-v3.patch
          119 kB
          Klaas Bosteels
        7. HADOOP-1722-v4.patch
          121 kB
          Klaas Bosteels
        8. HADOOP-1722-v4.patch
          119 kB
          Klaas Bosteels
        9. HADOOP-1722-v5.patch
          153 kB
          Klaas Bosteels
        10. HADOOP-1722-v6.patch
          153 kB
          Klaas Bosteels

        Issue Links

          Activity

            People

              klbostee Klaas Bosteels
              runping Runping Qi
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: