Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4574

Fix TotalOrderParitioner to work with non-WritableComparable key types

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.

      To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.

      That is, since we do not use the values at all (NullWritable), we may as well do:

        private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
            Configuration conf) throws IOException {
          …
          while ((key = (K) reader.next(key)) != null) {
            parts.add(key);
            key = ReflectionUtils.newInstance(keyClass, conf);
          }
          …
        }
      
      1. MAPREDUCE-4574.patch
        8 kB
        Harsh J
      2. MAPREDUCE-4574.patch
        9 kB
        Harsh J

        Activity

          People

          • Assignee:
            Harsh J
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development