[SPARK-5293] Enable Spark user applications to use different versions of Akka

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Shixiong Zhu
• Reporter: Reynold Xin
• Created: 17/Jan/15 02:36
• Updated: 23/Jan/16 05:22
• Resolved: 23/Jan/16 05:22

Description

A lot of Spark user applications are using (or want to use) Akka. Akka as a whole can contribute great architectural simplicity and uniformity. However, because Spark depends on Akka, it is not possible for users to rely on different versions, and we have received many requests in the past asking for help about this specific issue. For example, Spark Streaming might be used as the receiver of Akka messages - but our dependency on Akka requires the upstream Akka actors to also use the identical version of Akka.

Since our usage of Akka is limited (mainly for RPC and single-threaded event loop), we can replace it with alternative RPC implementations and a common event loop in Spark.

Issue Links

• blocks: SPARK-5487 Dockerfile to build spark's custom akka.
• is related to: SPARK-2949 SparkContext does not fate-share with ActorSystem
• relates to: SPARK-11095 Simplify Netty RPC implementation by using a separate thread pool for each endpoint