Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5293

Enable Spark user applications to use different versions of Akka

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      A lot of Spark user applications are using (or want to use) Akka. Akka as a whole can contribute great architectural simplicity and uniformity. However, because Spark depends on Akka, it is not possible for users to rely on different versions, and we have received many requests in the past asking for help about this specific issue. For example, Spark Streaming might be used as the receiver of Akka messages - but our dependency on Akka requires the upstream Akka actors to also use the identical version of Akka.

      Since our usage of Akka is limited (mainly for RPC and single-threaded event loop), we can replace it with alternative RPC implementations and a common event loop in Spark.

        Attachments

        Issue Links

        1.
        Add EventLoop and change DAGScheduler to an EventLoop Sub-task Resolved Shixiong Zhu Actions
        2.
        Standardize internal RPC interface Sub-task Resolved Shixiong Zhu Actions
        3.
        Replace direct use of Akka with Spark RPC interface Sub-task Resolved Shixiong Zhu Actions
        4.
        Provide an alternative RPC implementation based on the network transport module Sub-task Resolved Shixiong Zhu Actions
        5.
        Remove Akka systemName from Spark Sub-task Resolved Shixiong Zhu Actions
        6.
        Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoop Sub-task Resolved Shixiong Zhu Actions
        7.
        Update ReceiverTrackerActor to use the new Rpc interface Sub-task Resolved Shixiong Zhu Actions
        8.
        Deprecate configurations for "askWithReply" and use new configuration names Sub-task Resolved Shixiong Zhu Actions
        9.
        Move calling `TaskScheduler.executorHeartbeatReceived` to another thread to avoid blocking the Akka thread pool Sub-task Resolved Shixiong Zhu Actions
        10.
        Rename RPC askWithReply -> askWithReply, sendWithReply -> ask Sub-task Resolved Reynold Xin Actions
        11.
        Use the new RPC implementation by default Sub-task Resolved Shixiong Zhu Actions
        12.
        Remove "actorSystem" from SparkEnv Sub-task Resolved Unassigned Actions
        13.
        Move "AkkaRpcEnv" to a separate project Sub-task Resolved Unassigned Actions
        14.
        Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext Sub-task Resolved Shixiong Zhu Actions
        15.
        Remove "StreamingContext.actorStream" Sub-task Resolved Shixiong Zhu Actions
        16.
        Remove AkkaRpcEnv and remove Akka from the dependencies of Core Sub-task Resolved Shixiong Zhu Actions
        17.
        Deprecate the developer api SparkEnv.actorSystem Sub-task Resolved Ilya Ganelin Actions
        18.
        Remove the developer api SparkEnv.actorSystem and AkkaUtils Sub-task Resolved Shixiong Zhu Actions
        19.
        Remove 'Actor' from the comments Sub-task Resolved Nan Zhu Actions
        20.
        Remove akka dependency from SecurityManager Sub-task Resolved Marcelo Masiero Vanzin Actions
        21.
        Post-hoc review Netty based RPC implementation - round 1 Sub-task Resolved Reynold Xin Actions
        22.
        Post-hoc review Netty based RPC implementation - round 2 Sub-task Resolved Reynold Xin Actions
        23.
        Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive Sub-task Resolved Shixiong Zhu Actions
        24.
        RPC message ordering is not guaranteed Sub-task Resolved Shixiong Zhu Actions
        25.
        Add Outbox to cache the sending messages to resolve the message disorder issue Sub-task Closed Unassigned Actions

          Activity

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              rxin Reynold Xin

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment