Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5293

Enable Spark user applications to use different versions of Akka

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      A lot of Spark user applications are using (or want to use) Akka. Akka as a whole can contribute great architectural simplicity and uniformity. However, because Spark depends on Akka, it is not possible for users to rely on different versions, and we have received many requests in the past asking for help about this specific issue. For example, Spark Streaming might be used as the receiver of Akka messages - but our dependency on Akka requires the upstream Akka actors to also use the identical version of Akka.

      Since our usage of Akka is limited (mainly for RPC and single-threaded event loop), we can replace it with alternative RPC implementations and a common event loop in Spark.

        Issue Links

        1.
        Add EventLoop and change DAGScheduler to an EventLoop Sub-task Resolved Shixiong Zhu
         
        2.
        Standardize internal RPC interface Sub-task Resolved Shixiong Zhu
         
        3.
        Replace direct use of Akka with Spark RPC interface Sub-task Resolved Shixiong Zhu
         
        4.
        Provide an alternative RPC implementation based on the network transport module Sub-task Resolved Shixiong Zhu
         
        5.
        Remove Akka systemName from Spark Sub-task Resolved Shixiong Zhu
         
        6.
        Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoop Sub-task Resolved Shixiong Zhu
         
        7.
        Update ReceiverTrackerActor to use the new Rpc interface Sub-task Resolved Shixiong Zhu
         
        8.
        Deprecate configurations for "askWithReply" and use new configuration names Sub-task Resolved Shixiong Zhu
         
        9.
        Move calling `TaskScheduler.executorHeartbeatReceived` to another thread to avoid blocking the Akka thread pool Sub-task Resolved Shixiong Zhu
         
        10.
        Rename RPC askWithReply -> askWithReply, sendWithReply -> ask Sub-task Resolved Reynold Xin
         
        11.
        Use the new RPC implementation by default Sub-task Resolved Shixiong Zhu
         
        12.
        Remove "actorSystem" from SparkEnv Sub-task Resolved Unassigned
         
        13.
        Move "AkkaRpcEnv" to a separate project Sub-task Resolved Unassigned
         
        14.
        Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext Sub-task Resolved Shixiong Zhu
         
        15.
        Remove "StreamingContext.actorStream" Sub-task Resolved Shixiong Zhu
         
        16.
        Remove AkkaRpcEnv and remove Akka from the dependencies of Core Sub-task Resolved Shixiong Zhu
         
        17.
        Deprecate the developer api SparkEnv.actorSystem Sub-task Resolved Ilya Ganelin
         
        18.
        Remove the developer api SparkEnv.actorSystem and AkkaUtils Sub-task Resolved Shixiong Zhu
         
        19.
        Remove 'Actor' from the comments Sub-task Resolved Nan Zhu
         
        20.
        Remove akka dependency from SecurityManager Sub-task Resolved Marcelo Vanzin
         
        21.
        Post-hoc review Netty based RPC implementation - round 1 Sub-task Resolved Reynold Xin
         
        22.
        Post-hoc review Netty based RPC implementation - round 2 Sub-task Resolved Reynold Xin
         
        23.
        Add connection established callback to lower level RPC layer so we don't need to check for new connections in NettyRpcHandler.receive Sub-task Resolved Shixiong Zhu
         
        24.
        RPC message ordering is not guaranteed Sub-task Resolved Shixiong Zhu
         
        25.
        Add Outbox to cache the sending messages to resolve the message disorder issue Sub-task Closed Unassigned
         

          Activity

          Hide
          CodingCat Nan Zhu added a comment -

          shall we make this JIRA as an umbrella task, so that the other JIRAs like SPARK-5214 can be associated to here?

          Show
          CodingCat Nan Zhu added a comment - shall we make this JIRA as an umbrella task, so that the other JIRAs like SPARK-5214 can be associated to here?
          Hide
          vanzin Marcelo Vanzin added a comment -

          I converted this bug to an umbrella so that it's easier to track sub-tasks. Hope that's ok for everybody.

          Show
          vanzin Marcelo Vanzin added a comment - I converted this bug to an umbrella so that it's easier to track sub-tasks. Hope that's ok for everybody.

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              rxin Reynold Xin
            • Votes:
              9 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development