Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5293

Enable Spark user applications to use different versions of Akka

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • Spark Core
    • None

    Description

      A lot of Spark user applications are using (or want to use) Akka. Akka as a whole can contribute great architectural simplicity and uniformity. However, because Spark depends on Akka, it is not possible for users to rely on different versions, and we have received many requests in the past asking for help about this specific issue. For example, Spark Streaming might be used as the receiver of Akka messages - but our dependency on Akka requires the upstream Akka actors to also use the identical version of Akka.

      Since our usage of Akka is limited (mainly for RPC and single-threaded event loop), we can replace it with alternative RPC implementations and a common event loop in Spark.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-5487 Dockerfile to build spark's custom akka.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-2949 SparkContext does not fate-share with ActorSystem

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-11095 Simplify Netty RPC implementation by using a separate thread pool for each endpoint

          • Major - Major loss of function.
          • Resolved
          1.
          		 Add EventLoop and change DAGScheduler to an EventLoop Sub-task Resolved Shixiong Zhu
          2.
          		 Standardize internal RPC interface Sub-task Resolved Shixiong Zhu
          3.
          		 Replace direct use of Akka with Spark RPC interface Sub-task Resolved Shixiong Zhu
          4.
          		 Provide an alternative RPC implementation based on the network transport module Sub-task Resolved Shixiong Zhu
          5.
          		 Remove Akka systemName from Spark Sub-task Resolved Shixiong Zhu
          6.
          		 Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoop Sub-task Resolved Shixiong Zhu
          7.
          		 Update ReceiverTrackerActor to use the new Rpc interface Sub-task Resolved Shixiong Zhu
          8.
          		 Deprecate configurations for "askWithReply" and use new configuration names Sub-task Resolved Shixiong Zhu
          9.
          		 Move calling `TaskScheduler.executorHeartbeatReceived` to another thread to avoid blocking the Akka thread pool Sub-task Resolved Shixiong Zhu
          10.
          		 Rename RPC askWithReply -> askWithReply, sendWithReply -> ask Sub-task Resolved Reynold Xin
          11.
          		 Use the new RPC implementation by default Sub-task Resolved Shixiong Zhu
          12.
          		 Remove "actorSystem" from SparkEnv Sub-task Resolved Unassigned
          13.
          		 Move "AkkaRpcEnv" to a separate project Sub-task Resolved Unassigned
          14.
          		 Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext Sub-task Resolved Shixiong Zhu
          15.
          		 Remove "StreamingContext.actorStream" Sub-task Resolved Shixiong Zhu
          16.
          		 Remove AkkaRpcEnv and remove Akka from the dependencies of Core Sub-task Resolved Shixiong Zhu
          17.
          		 Deprecate the developer api SparkEnv.actorSystem Sub-task Resolved Ilya Ganelin
          18.
          		 Remove the developer api SparkEnv.actorSystem and AkkaUtils Sub-task Resolved Shixiong Zhu
          19.
          		 Remove 'Actor' from the comments Sub-task Resolved Nan Zhu
          20.
          		 Remove akka dependency from SecurityManager Sub-task Resolved Marcelo Masiero Vanzin
          21.
          		 Post-hoc review Netty based RPC implementation - round 1 Sub-task Resolved Reynold Xin
          22.
          		 Post-hoc review Netty based RPC implementation - round 2 Sub-task Resolved Reynold Xin
          23.
          		 Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive Sub-task Resolved Shixiong Zhu
          24.
          		 RPC message ordering is not guaranteed Sub-task Resolved Shixiong Zhu
          25.
          		 Add Outbox to cache the sending messages to resolve the message disorder issue Sub-task Closed Unassigned

          Activity

            People

              zsxwing Shixiong Zhu
              rxin Reynold Xin
              Votes:
              9 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: