Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1287

Partitioner class not used in TableMapReduceUtil.initTableReduceJob()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.2, 0.20.0
    • None
    • None

    Description

      Upon checking the available utility methods in TableMapReduceUtil I came across this code

        public static void initTableReduceJob(String table,
          Class<? extends TableReduce> reducer, JobConf job, Class partitioner)
        throws IOException {
          job.setOutputFormat(TableOutputFormat.class);
          job.setReducerClass(reducer);
          job.set(TableOutputFormat.OUTPUT_TABLE, table);
          job.setOutputKeyClass(ImmutableBytesWritable.class);
          job.setOutputValueClass(BatchUpdate.class);
          if (partitioner != null) {
            job.setPartitionerClass(HRegionPartitioner.class);
            HTable outputTable = new HTable(new HBaseConfiguration(job), table);
            int regions = outputTable.getRegionsInfo().size();
            if (job.getNumReduceTasks() > regions){
          	job.setNumReduceTasks(outputTable.getRegionsInfo().size());
            }
          }
        }
      

      It seems though as it should be

          if (partitioner != null) {
            job.setPartitionerClass(partitioner);
      

      and the provided HRegionPartitioner can be handed in to that call or a custom one can be provided.

      Attachments

        1. 1287-6.patch.txt
          11 kB
          Billy Pearson
        2. 1287-5.patch
          6 kB
          Lars George
        3. 1287-4-patch.txt
          1 kB
          Billy Pearson
        4. 1287-3-patch.txt
          1 kB
          Billy Pearson
        5. 1287-2.patch
          3 kB
          Lars George
        6. 1287.patch
          0.7 kB
          Lars George

        Activity

          People

            viper799 Billy Pearson
            larsgeorge Lars George
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: