[PIG-5297] Yarn-client mode doesn't work with Spark 2

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: spark
• Assignee: Ádám Szita
• Reporter: Ádám Szita
• Created: 01/Sep/17 13:17
• Updated: 14/Sep/17 15:52

Description

When running tests in yarn-client mode that were built with Spark 2 I'm getting the following exception:

Caused by: java.lang.IllegalStateException: Library directory '...../pig/assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
	at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
	at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
	at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:558)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:882)

After overcoming this with symlinks and setting SPARK_HOME I hit another issue:

Caused by: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V
	at org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty(FileSegmentManagedBuffer.java:133)
	at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:58)
	at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:33)
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)

I believe this will be an incompatibility between netty-all versions required by hadoop and spark..