Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
If the number desired partition is a multiples of 31, HashPartitioner doesn't make desired number of partitions. I tested with the following code.
Random rand = new Random(); String[][] data = new String[1000][]; for (int i = 0; i < 1000; i++) { data[i] = new String[]{ String.valueOf(rand.nextInt(1000)), String.valueOf(rand.nextInt(1000)), String.valueOf(rand.nextInt(1000))}; } int[] testNumPartitions = new int[]{31, 62, 124, 32, 63, 125}; for (int index = 0; index < testNumPartitions.length; index++) { Set<Integer> ids = new TreeSet<Integer>(); for (int i = 0; i < data.length; i++) { VTuple vtuple = new VTuple(new Datum[]{new TextDatum(data[i][0]), new TextDatum(data[i][1]), new TextDatum(data[i][2])}); int hashValue = vtuple.hashCode(); int partitionId = (hashValue & Integer.MAX_VALUE) % testNumPartitions[index]; ids.add(partitionId); } System.out.println("Expected partition num: " + testNumPartitions[index] + ", real partition num:" + ids.size()); }
Expected partition num: 31, real partition num:10 Expected partition num: 62, real partition num:20 Expected partition num: 124, real partition num:40 Expected partition num: 32, real partition num:32 Expected partition num: 63, real partition num:63 Expected partition num: 125, real partition num:125