[SPARK-5487] Dockerfile to build spark's custom akka.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.2.0
• Component/s: Build
• Assignee: Unassigned
• Reporter: jay vyas
• Created: 29/Jan/15 19:55
• Updated: 08/Jan/16 06:47
• Resolved: 08/Jan/16 06:47

Description

Building spark's custom shaed akka version is tricky. The code is in https://github.com/pwendell/akka/ (branch = 2.2.3-shaded-proto) , however, when attempting to build, I receive some strange errors.

I've attempted to fork off of a Dockerfile for SBT 0.12.4, which I'll attach in a snippet just as an example of what we might want to facilitate building the spark specific akka until SPARK-5293 is completed.

[info] Compiling 6 Scala sources and 1 Java source to /tmp/akka/akka-multi-node-testkit/target/classes...
[warn] Class com.google.protobuf.MessageLite not found - continuing with a stub.
[error] error while loading ProtobufDecoder, class file '/root/.ivy2/cache/io.netty/netty/bundles/netty-3.6.6.Final.jar(org/jboss/netty/handler/codec/protobuf/ProtobufDecoder.class)' is broken
[error] (class java.lang.NullPointerException/null)
[error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testconductor/RemoteConnection.scala:24: org.jboss.netty.handler.codec.protobuf.ProtobufDecoder does not have a constructor
[error]     val proto = List(new ProtobufEncoder, new ProtobufDecoder(TestConductorProtocol.Wrapper.getDefaultInstance))
[error]                                           ^
[error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:267: value await is not a member of scala.concurrent.Future[Iterable[akka.remote.testconductor.RoleName]]
[error]  Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type
[error]           testConductor.getNodes.await.filterNot(_ == myself).isEmpty
[error]                                  ^
[error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:354: value await is not a member of scala.concurrent.Future[akka.actor.Address]
[error]  Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type
[error]   def node(role: RoleName): ActorPath = RootActorPath(testConductor.getAddressFor(role).await)
[error]                                                                                         ^
[warn] one warning found
[error] four errors found
[info] Updating {file:/tmp/akka/}akka-docs...
[info] Done updating.
[info] Updating {file:/tmp/akka/}akka-contrib...
[info] Done updating.
[info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-core...
[info] Done updating.
[info] Compiling 17 Scala sources to /tmp/akka/akka-cluster/target/classes...
[error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:59: type mismatch;
[error]  found   : akka.cluster.protobuf.msg.GossipEnvelope
[error]  required: com.google.protobuf_spark.MessageLite
[error]       case m: GossipEnvelope ? compress(gossipEnvelopeToProto(m))
[error]                                                              ^
[error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:61: type mismatch;
[error]  found   : akka.cluster.protobuf.msg.MetricsGossipEnvelope
[error]  required: com.google.protobuf_spark.MessageLite
[error]       case m: MetricsGossipEnvelope ? compress(metricsGossipEnvelopeToProto(m))
[error]                                                                            ^
[error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:63: type mismatch;
[error]  found   : akka.cluster.protobuf.msg.Welcome
[error]  required: com.google.protobuf_spark.MessageLite
[error]       case InternalClusterAction.Welcome(from, gossip) ? compress(msg.Welcome(uniqueAddressToProto(from), gossipToProto(gossip)))
[error]                                                                              ^
[error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:257: type mismatch;
[error]  found   : com.google.protobuf_spark.ByteString
[error]  required: com.google.protobuf.ByteString
[error]           msg.NodeMetrics.Number(msg.NodeMetrics.NumberType.Serialized, None, None, Some(ByteString.copyFrom(bos.toByteArray)))
[error]                                                                                                             ^
[error] four errors found
[info] Updating {file:/tmp/akka/}akka-sample-cluster...
[info] Done updating.
[info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-integration...
[info] Done updating.
[error] (akka-multi-node-testkit/compile:compile) Compilation failed
[error] (akka-cluster/compile:compile) Compilation failed
[error] Total time: 21 s, completed Jan 29, 2015 5:51:14 PM
root@c938961f45f1:/tmp/akka# 

Issue Links

• is blocked by: SPARK-5293 Enable Spark user applications to use different versions of Akka