Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1462

Enable context-specific and stateful serializers in MapReduce

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None

      Description

      Although the current serializer framework is powerful, within the context of a job it is limited to picking a single serializer for a given class. Additionally, Avro generic serialization can make use of additional configuration/state such as the schema. (Most other serialization frameworks including Writable, Jute/Record IO, Thrift, Avro Specific, and Protocol Buffers only need the object's class name to deserialize the object.)

      With the goal of keeping the easy things easy and maintaining backwards compatibility, we should be able to allow applications to use context specific (eg. map output key) serializers in addition to the current type based ones that handle the majority of the cases. Furthermore, we should be able to support serializer specific configuration/metadata in a type safe manor without cluttering up the base API with a lot of new methods that will confuse new users.

      1. h-1462.patch
        14 kB
        Owen O'Malley
      2. MAPREDUCE-1462-common.patch
        5 kB
        Tom White
      3. MAPREDUCE-1462-mr.patch
        77 kB
        Tom White

        Issue Links

          Activity

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              2 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:

                Development