[HCATALOG-494] MultiOutputFormat in 0.23 fails to setAliasConf() correctly.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.1
• Fix Version/s: 0.4.1
• Component/s: mapreduce

Environment

Hadoop-0.23.1

• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 07/Sep/12 01:43
• Updated: 20/Sep/12 19:12
• Resolved: 20/Sep/12 19:12

Description

When using MultiOutputFormat (in Sta^H^H^H a super-duper top secret project that shall not be named), against Hadoop 0.23.1, one sees the following:

<exception>
java.lang.StringIndexOutOfBoundsException: String index out of range: -2
at java.lang.AbstractStringBuilder.delete(AbstractStringBuilder.java:698)
at java.lang.StringBuilder.delete(StringBuilder.java:255)
at org.apache.hcatalog.mapreduce.MultiOutputFormat.setAliasConf(MultiOutputFormat.java:291)
at org.apache.hcatalog.mapreduce.MultiOutputFormat.checkOutputSpecs(MultiOutputFormat.java:224)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
...
</exception>

The error boils down to a change in the semantics of JobContext::JobContext(Configuration, JobID). While in .20, the Config was cloned, in .23 the Config is adopted (iff it's a JobConf). This causes the same Configuration instance to be written-to for different tables in the same job.

An easy way to deal with this is to clone the Configuration before creating a JobContextImpl.

A patch is on its way.