[HIVE-7387] Guava version conflict between hadoop and spark [Spark-Branch]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: Spark
• Assignee: Chengxiang Li
• Reporter: Chengxiang Li
• Created: 11/Jul/14 03:14
• Updated: 28/Jan/15 14:35
• Resolved: 24/Oct/14 17:58

Description

The guava conflict happens in hive driver compile stage, as in the follow exception stacktrace, conflict happens while initiate spark RDD in SparkClient, hive driver take both guava 11 from hadoop classpath and spark assembly jar which contains guava 14 classes in its classpath, spark invoked HashFunction.hasInt which method does not exists in guava 11 version, obvious the guava 11 version HashFunction is loaded into the JVM, which lead to a NoSuchMethodError during initiate spark RDD.

java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode;
	at org.apache.spark.util.collection.OpenHashSet.org$apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261)
	at org.apache.spark.util.collection.OpenHashSet$mcI$sp.getPos$mcI$sp(OpenHashSet.scala:165)
	at org.apache.spark.util.collection.OpenHashSet$mcI$sp.contains$mcI$sp(OpenHashSet.scala:102)
	at org.apache.spark.util.SizeEstimator$$anonfun$visitArray$2.apply$mcVI$sp(SizeEstimator.scala:214)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
	at org.apache.spark.util.SizeEstimator$.visitArray(SizeEstimator.scala:210)
	at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:169)
	at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:161)
	at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:155)
	at org.apache.spark.storage.MemoryStore.putValues(MemoryStore.scala:75)
	at org.apache.spark.storage.MemoryStore.putValues(MemoryStore.scala:92)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:661)
	at org.apache.spark.storage.BlockManager.put(BlockManager.scala:546)
	at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:812)
	at org.apache.spark.broadcast.HttpBroadcast.<init>(HttpBroadcast.scala:52)
	at org.apache.spark.broadcast.HttpBroadcastFactory.newBroadcast(HttpBroadcastFactory.scala:35)
	at org.apache.spark.broadcast.HttpBroadcastFactory.newBroadcast(HttpBroadcastFactory.scala:29)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:776)
	at org.apache.spark.rdd.HadoopRDD.<init>(HadoopRDD.scala:112)
	at org.apache.spark.SparkContext.hadoopRDD(SparkContext.scala:527)
	at org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:307)
	at org.apache.hadoop.hive.ql.exec.spark.SparkClient.createRDD(SparkClient.java:204)
	at org.apache.hadoop.hive.ql.exec.spark.SparkClient.execute(SparkClient.java:167)
	at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:32)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:159)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)

NO PRECOMMIT TESTS. This is for spark branch only.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-8878 Downgrade guava version to be consistent with Hive and the rest of Hadoop [Spark Branch]
• is related to: HIVE-7437 Check if servlet-api and jetty module in Spark library are an issue for hive-spark integration [Spark Branch]
• relates to: HIVE-8854 Guava dependency conflict between hive driver and remote spark context[Spark Branch]
• requires: SPARK-2420 Dependency changes for compatibility with Hive