[SPARK-3965] Spark assembly for hadoop2 contains avro-mapred for hadoop1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.2, 1.1.0, 1.2.0
• Component/s: Input/Output

Environment

hadoop2, HDP2.1

• Assignee: Unassigned
• Reporter: David Jacot
• Created: 16/Oct/14 05:36
• Updated: 08/Feb/15 13:32
• Resolved: 08/Feb/15 13:32

Description

When building Spark assembly for hadoop2, org.apache.avro:avro-mapred for hadoop1 is picked and added to the assembly which leads to following exception at runtime.

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
        at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47)
        at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:111)
...

The patch for SPARK-3039 works well at compile time but artefact's classifier is not applied when assembly is built. I'm not a maven expert but I don't think that classifiers are applied on transitive dependencies.

Issue Links

• duplicates: SPARK-3039 Spark assembly for new hadoop API (hadoop 2) contains avro-mapred for hadoop 1 API
• links to: [Github] Pull Request #2822 (dajac)