Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5293

Enable Spark user applications to use different versions of Akka

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      A lot of Spark user applications are using (or want to use) Akka. Akka as a whole can contribute great architectural simplicity and uniformity. However, because Spark depends on Akka, it is not possible for users to rely on different versions, and we have received many requests in the past asking for help about this specific issue. For example, Spark Streaming might be used as the receiver of Akka messages - but our dependency on Akka requires the upstream Akka actors to also use the identical version of Akka.

      Since our usage of Akka is limited (mainly for RPC and single-threaded event loop), we can replace it with alternative RPC implementations and a common event loop in Spark.

        Attachments

          Issue Links

          1.
          Add EventLoop and change DAGScheduler to an EventLoop Sub-task Resolved Shixiong Zhu
          2.
          Standardize internal RPC interface Sub-task Resolved Shixiong Zhu
          3.
          Replace direct use of Akka with Spark RPC interface Sub-task Resolved Shixiong Zhu
          4.
          Provide an alternative RPC implementation based on the network transport module Sub-task Resolved Shixiong Zhu
          5.
          Remove Akka systemName from Spark Sub-task Resolved Shixiong Zhu
          6.
          Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoop Sub-task Resolved Shixiong Zhu
          7.
          Update ReceiverTrackerActor to use the new Rpc interface Sub-task Resolved Shixiong Zhu
          8.
          Deprecate configurations for "askWithReply" and use new configuration names Sub-task Resolved Shixiong Zhu
          9.
          Move calling `TaskScheduler.executorHeartbeatReceived` to another thread to avoid blocking the Akka thread pool Sub-task Resolved Shixiong Zhu
          10.
          Rename RPC askWithReply -> askWithReply, sendWithReply -> ask Sub-task Resolved Reynold Xin
          11.
          Use the new RPC implementation by default Sub-task Resolved Shixiong Zhu
          12.
          Remove "actorSystem" from SparkEnv Sub-task Resolved Unassigned
          13.
          Move "AkkaRpcEnv" to a separate project Sub-task Resolved Unassigned
          14.
          Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext Sub-task Resolved Shixiong Zhu
          15.
          Remove "StreamingContext.actorStream" Sub-task Resolved Shixiong Zhu
          16.
          Remove AkkaRpcEnv and remove Akka from the dependencies of Core Sub-task Resolved Shixiong Zhu
          17.
          Deprecate the developer api SparkEnv.actorSystem Sub-task Resolved Ilya Ganelin
          18.
          Remove the developer api SparkEnv.actorSystem and AkkaUtils Sub-task Resolved Shixiong Zhu
          19.
          Remove 'Actor' from the comments Sub-task Resolved Nan Zhu
          20.
          Remove akka dependency from SecurityManager Sub-task Resolved Marcelo Vanzin
          21.
          Post-hoc review Netty based RPC implementation - round 1 Sub-task Resolved Reynold Xin
          22.
          Post-hoc review Netty based RPC implementation - round 2 Sub-task Resolved Reynold Xin
          23.
          Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive Sub-task Resolved Shixiong Zhu
          24.
          RPC message ordering is not guaranteed Sub-task Resolved Shixiong Zhu
          25.
          Add Outbox to cache the sending messages to resolve the message disorder issue Sub-task Closed Unassigned

            Activity

              People

              • Assignee:
                zsxwing Shixiong Zhu
                Reporter:
                rxin Reynold Xin
              • Votes:
                9 Vote for this issue
                Watchers:
                38 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: