[AVRO-2138] org.apache.avro.mapreduce.AvroMultipleOutputs.write copies Configuration on every invocation of write

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.8.2
• Component/s: java
• Assignee: Unassigned
• Reporter: Stan Rosenberg
• Created: 30/Jan/18 23:39
• Updated: 30/Dec/18 04:09

Description

While profiling a spark job using AvroMultipleOutputs, I noticed that a great deal of time is wasted by copying (hadoop) Configuration.  Indeed this happens on every invocation of write: https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroMultipleOutputs.java#L437

After patching, I am seeing a speed-up of 2x and above in running time of the same job.