Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4346

Implement basic RPC abstraction

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: Runtime / Coordination
    • Labels:

      Description

      As part of refactoring of the cluster management, we can introduce a new RPC abstraction on top of our Akka-based distributed coordination.

      It should address the following issues:

      • Add type safety to the sender and receiver of messages. We want proper types methods to be called, rather than haveing generic message types and pattern matching everywhere. This is similar to typed actors.
      • Make the message receivers testable without involving actors, i.e. the methods should be callable directly. When used with other component, the receiver will be wrapped in an actor that calls the methods based on received messages.
      • We want to keep the paradigm of single-threaded execution per "actor"

      There is some basic code layout in the following branch and commit:

      https://github.com/apache/flink/tree/flip-6/flink-runtime/src/main/java/org/apache/flink/runtime/rpc

        Attachments

          Issue Links

          1.
          Introduce Flink's own future abstraction Sub-task Resolved Till Rohrmann
          2.
          Auto generate message sender classes via Java Proxies Sub-task Resolved Till Rohrmann
          3.
          Eagerly initialize RrcProtocol members Sub-task Closed Stephan Ewen
          4.
          Buffer rpc calls until RpcEndpoint is properly started Sub-task Closed Till Rohrmann
          5.
          Check parameters for serializability before sending a remote RpcInvocation message Sub-task Closed Till Rohrmann
          6.
          Add a "scheduleRunAsync()" feature to the RpcEndpoint Sub-task Closed Stephan Ewen
          7.
          Add as way to assert that code runs in the RpcEndpoint's Main Thread Sub-task Closed Stephan Ewen
          8.
          Add throws clause verification to RpcCompletenessTest Sub-task Closed Till Rohrmann
          9.
          Make RPC Service Thread Safe Sub-task Closed Stephan Ewen
          10.
          RPC proxy classloading should use Flink class' classloader Sub-task Closed Stephan Ewen
          11.
          Remove restriction on RpcService.getAddress Sub-task Resolved Till Rohrmann
          12.
          Add a testing RPC service Sub-task Resolved Stephan Ewen
          13.
          Add support in RpcCompletenessTest for inheritance of RpcGateway and RpcEndpoint Sub-task Closed Wenlong Lyu
          14.
          Throw exception when remote connection cannot be resolved Sub-task Closed Till Rohrmann
          15.
          Mark runAsync and scheduleAsync methods protected Sub-task Closed Till Rohrmann
          16.
          when call connect method in AkkaRpcService using same address and same rpc gateway class, the returned gateways are equal with respect to equals and hashCode Sub-task Closed Jing Zhang
          17.
          Check that the RpcEndpoint supports the specified RpcGateway Sub-task Closed Till Rohrmann
          18.
          Port existing code to use Flink's future abstraction Sub-task Closed Till Rohrmann
          19.
          Allow RpcService to execute Callables in the RpcService executor Sub-task Closed Till Rohrmann
          20.
          Add getAddress method to RpcService Sub-task Closed Till Rohrmann
          21.
          Add wait for termination function to RpcEndpoints Sub-task Closed Till Rohrmann
          22.
          RpcCompletenessTest: Add support for type arguments and subclasses Sub-task Closed Maximilian Michels
          23.
          Let RpcEndpoint.start/shutDown throw exceptions Sub-task Closed Till Rohrmann
          24.
          Remove stashing from AkkaRpcActor Sub-task Closed Till Rohrmann
          25.
          Properly unpack thrown exceptions in RPC methods Sub-task Closed Stephan Ewen
          26.
          Let the RPCService provide a ScheduledExecutorService Sub-task Closed Till Rohrmann
          27.
          Let RpcService.scheduleRunnable return ScheduledFuture Sub-task Closed Till Rohrmann
          28.
          Allow to specify endpoint names Sub-task Closed Till Rohrmann
          29.
          Create SerialMainThreadValidatorUtil to support TestingSerialRpcService Sub-task Closed Till Rohrmann
          30.
          Remove TestingSerialRpcService Sub-task Closed Till Rohrmann
          31.
          Support fencing tokens to filter out outdated messages Sub-task Closed Till Rohrmann
          32.
          Add getPort method to RpcService Sub-task Closed Till Rohrmann
          33.
          Add callback for proper RpcEndpoint shut down Sub-task Closed Till Rohrmann
          34.
          Let RpcEndpoint directly implement RpcGateways Sub-task Closed Till Rohrmann
          35.
          Complete termination future after actor has been stopped. Sub-task Closed Till Rohrmann
          36.
          Simplify termination future completion Sub-task Closed Till Rohrmann
          37.
          Shut down AkkaRpcActors with PoisonPill Sub-task Closed Till Rohrmann
          38.
          Change termination future type of RpcEndpoint to Void Sub-task Closed Till Rohrmann
          39.
          Allow RpcEndpoint#postStop to complete asynchronously Sub-task Closed Till Rohrmann
          40.
          Make AkkaRpcService#stopService non-blocking Sub-task Closed Till Rohrmann
          41.
          Add UnfencedMainThreadExecutor to FencedRpcEndpoint Sub-task Closed Till Rohrmann
          42.
          Add support for scheduleRunAsync for FencedRpcEndpoints Sub-task Closed Till Rohrmann
          43.
          Add version information to remote rpc messages Sub-task Closed Till Rohrmann

            Activity

              People

              • Assignee:
                trohrmann Till Rohrmann
                Reporter:
                sewen Stephan Ewen
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: