Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1126

shuffle should use serialization to get comparator

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the key comparator is defined as a Java class. Instead we should use the Serialization API to create key comparators. This would permit, e.g., Avro-based comparators to be used, permitting efficient sorting of complex data types without having to write a RawComparator in Java.

        Attachments

        1. MAPREDUCE-1126.patch
          90 kB
          Aaron Kimball
        2. MAPREDUCE-1126.2.patch
          107 kB
          Aaron Kimball
        3. MAPREDUCE-1126.3.patch
          107 kB
          Aaron Kimball
        4. MAPREDUCE-1126.4.patch
          108 kB
          Aaron Kimball
        5. MAPREDUCE-1126.5.patch
          112 kB
          Aaron Kimball
        6. MAPREDUCE-1126.6.patch
          120 kB
          Aaron Kimball
        7. MAPREDUCE-1126.patch
          75 kB
          Tom White
        8. m-1126-2.patch
          2 kB
          Owen O'Malley
        9. m-1126-3.patch
          2 kB
          Owen O'Malley

          Issue Links

            Activity

              People

              • Assignee:
                kimballa Aaron Kimball
                Reporter:
                cutting Doug Cutting
              • Votes:
                0 Vote for this issue
                Watchers:
                39 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: