[FLINK-7040] Flip-6 client-cluster communication

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Deployment / Mesos, Runtime / Coordination
• Labels: flip-6
• Assignee: Chesnay Schepler
• Reporter: Till Rohrmann
• Created: 29/Jun/17 12:30
• Updated: 29/Jan/21 10:33
• Resolved: 29/Jan/21 10:33

Description

With the new Flip-6 architecture, the client will communicate with the cluster in a RESTful manner.

The cluster shall support the following REST calls:

• List jobs (GET): Get list of all running jobs on the cluster
• Submit job (POST): Submit a job to the cluster (only supported in session mode)
• Lookup job leader (GET): Gets the JM leader for the given job
• Get job status (GET): Get the status of an executed job (and maybe the JobExecutionResult)
• Cancel job (PUT): Cancel the given job
• Stop job (PUT): Stops the given job
• Take savepoint (POST): Take savepoint for given job (How to return the savepoint under which the savepoint was stored? Maybe always having to specify a path)
• Get KV state (GET): Gets the KV state for the given job and key (Queryable state)
• Poll/subscribe to notifications for job (GET, WebSocket): Polls new notifications from the execution of the given job/Opens WebSocket to receive notifications

The first four REST calls will be served by the REST endpoint running in the application master/cluster entrypoint. The other calls will be served by a REST endpoint running along side to the JobManager.

Detailed information about different implementations and their pros and cons can be found in this document:
https://docs.google.com/document/d/1eIX6FS9stwraRdSUgRSuLXC1sL7NAmxtuqIXe_jSi-k/edit?usp=sharing

The implementation will most likely be Netty based.

Issue Links

• contains: FLINK-6387 Flink UI support access log
• is a child of: FLINK-7073 Create RESTful JobManager endpoint
• links to: GitHub Pull Request #4569

Sub-Tasks

• Create RESTful client: Closed
• Create RESTful cluster endpoint: Closed
• Enrich ClientRestException with HttpResponseStatus: Closed
• Create WebSocket handler (server): Closed
• RedirectHandler should execute on the IO thread: Resolved
• Integrate savepoint command into REST client: Closed
• Integrate list command into REST client: Closed
• Integrate LIST command into REST client: Closed
• Don't use PipelineErrorHandler on the client: Closed
• Extend AbstractRestHandler to return response headers: Closed
• Create RestHandler for long running operations: Closed
• Expose JobExecutionResult via HTTP: Closed
• Retrieve ExecutionResult by REST polling: Resolved