[SPARK-3764] Invalid dependencies of artifacts in Maven Central Repository.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.1.0
• Component/s: Build
• Assignee: Unassigned Assign to me
• Reporter: Takuya Ueshin
• Created: 02/Oct/14 10:25
• Updated: 03/Oct/14 12:46
• Resolved: 03/Oct/14 12:46

Description

While testing my spark applications locally using spark artifacts downloaded from Maven Central, the following exception was thrown:

ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-2,5,main]
java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected
at org.apache.spark.sql.parquet.AppendingParquetOutputFormat.getDefaultWorkFile(ParquetTableOperations.scala:334)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:251)
at org.apache.spark.sql.parquet.InsertIntoParquetTable.org$apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:300)
at org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
at org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

This is because the hadoop class TaskAttemptContext is incompatible between hadoop-1 and hadoop-2.

I guess the spark artifacts in Maven Central were built against hadoop-2 with Maven, but the depending version of hadoop in pom.xml remains 1.0.4, so the hadoop version mismatch is happend.

FYI:
sbt seems to publish 'effective pom'-like pom file, so the dependencies are correctly resolved.

Issue Links

• is related to: SPARK-3771 AppendingParquetOutputFormat should use reflection to prevent from breaking binary-compatibility.