[FLINK-26370] Make Flink cluster communication asynchronous

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: kubernetes-operator-0.1.0
• Component/s: Kubernetes Operator
• Labels: pull-request-available
• Assignee: Sandor Kelemen
• Reporter: Gyula Fora
• Created: 25/Feb/22 11:18
• Updated: 27/Mar/22 13:16
• Resolved: 24/Mar/22 10:10

Description

In the current architecture calls to the flink clusters (through the rest client) are made synchronously from the reconcile loop. 

These calls often take a long time due to various (compeltely normal) reasons:
 - Cluster is not ready -> long call + timeoutexception
 - Operation takes a long time -> cancel/savepoint operations are often expected to take seconds/minutes

Both the observer and reconciler components make these calls.

We should come up with a way to avoid making these sync calls from the main loop while still preserving the logic of the operator.

Issue Links

• links to: GitHub Pull Request #44