Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4574

Fix TotalOrderParitioner to work with non-WritableComparable key types

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.

      To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.

      That is, since we do not use the values at all (NullWritable), we may as well do:

        private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
            Configuration conf) throws IOException {
          …
          while ((key = (K) reader.next(key)) != null) {
            parts.add(key);
            key = ReflectionUtils.newInstance(keyClass, conf);
          }
          …
        }
      
      1. MAPREDUCE-4574.patch
        9 kB
        Harsh J
      2. MAPREDUCE-4574.patch
        8 kB
        Harsh J

        Activity

        Harsh J created issue -
        Harsh J made changes -
        Field Original Value New Value
        Assignee Harsh J [ qwertymaniac ]
        Harsh J made changes -
        Assignee Harsh J [ qwertymaniac ]
        Harsh J made changes -
        Priority Major [ 3 ] Trivial [ 5 ]
        Harsh J made changes -
        Attachment MAPREDUCE-4574.patch [ 12542447 ]
        Harsh J made changes -
        Attachment MAPREDUCE-4574.patch [ 12542449 ]
        Harsh J made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Harsh J [ qwertymaniac ]
        Harsh J made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Harsh J
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development