Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4574

Fix TotalOrderParitioner to work with non-WritableComparable key types

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.

      To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.

      That is, since we do not use the values at all (NullWritable), we may as well do:

        private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
            Configuration conf) throws IOException {
          …
          while ((key = (K) reader.next(key)) != null) {
            parts.add(key);
            key = ReflectionUtils.newInstance(keyClass, conf);
          }
          …
        }
      

        Attachments

        1. MAPREDUCE-4574.patch
          9 kB
          Harsh J
        2. MAPREDUCE-4574.patch
          8 kB
          Harsh J

          Activity

            People

            • Assignee:
              qwertymaniac Harsh J
              Reporter:
              qwertymaniac Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: