[TEZ-4133] key class implements writableComparable and configurable use default configuration

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: wang qiang
• Reporter: wang qiang
• Created: 10/Mar/20 02:01
• Updated: 21/Jul/20 15:54
• Resolved: 21/Jul/20 15:54

Description

key class implments writableComparable and configurable ,setConf be invoked with a default configuration,it should  like mapreduce(org.apache.hadoop.mapred.JobConf#getOutputKeyComparator)

//org.apache.tez.runtime.library.common.ConfigUtils#getIntermediateOutputKeyComparator
public static <K> RawComparator<K> getIntermediateOutputKeyComparator(Configuration conf) {
  Class<? extends RawComparator> theClass = conf.getClass(
      TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
      RawComparator.class);
  if (theClass != null)
    return ReflectionUtils.newInstance(theClass, conf);
// aybe change to 
// return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
  return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass(
      WritableComparable.class));
}

public static <K> RawComparator<K> getIntermediateInputKeyComparator(Configuration conf) {
  Class<? extends RawComparator> theClass = conf.getClass(
      TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
      RawComparator.class);
  if (theClass != null)
    return ReflectionUtils.newInstance(theClass, conf);
// maybe change to 
// return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
  return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass(
      WritableComparable.class));
}



 

 
 
 
 
 
 
 
 
 
 
 

Issue Links

• is related to: HADOOP-10686 Writables are not always configured