Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Cannot Reproduce
-
0.6.0
-
None
Description
It's not possible to directly run a pipeline on the spark runner (for instance using mvn exec:java. It fails with:
[appclient-register-master-threadpool-0] INFO org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to master spark://10.200.118.197:7077... [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted; java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted; at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73) at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101) at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090) at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) at java.lang.Thread.run(Thread.java:745) [appclient-register-master-threadpool-0] WARN org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect to master 10.200.118.197:7077 java.io.IOException: Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted; at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239) at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129) at io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:845) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101) at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090) at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted; at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73) at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) ... 15 more
It looks like a conflict between the Netty version used in Spark and the one in Beam (just guessing).
The workaround is to use spark-submit.
Attachments
Issue Links
1.
|
Support adding jars to spark runner | Open | Unassigned | |
2.
|
Document use of adding jars to spark runner | Open | Unassigned |