Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4574

Fix TotalOrderParitioner to work with non-WritableComparable key types

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.0.0-alpha
    • 3.0.0-alpha1
    • client
    • None
    • Reviewed

    Description

      The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.

      To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.

      That is, since we do not use the values at all (NullWritable), we may as well do:

        private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
            Configuration conf) throws IOException {
          …
          while ((key = (K) reader.next(key)) != null) {
            parts.add(key);
            key = ReflectionUtils.newInstance(keyClass, conf);
          }
          …
        }
      

      Attachments

        1. MAPREDUCE-4574.patch
          9 kB
          Harsh J
        2. MAPREDUCE-4574.patch
          8 kB
          Harsh J

        Activity

          People

            qwertymaniac Harsh J
            qwertymaniac Harsh J
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: