[BEAM-14449] Support cluster provisioning when using Flink on Dataproc

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.40.0
• Component/s: runner-py-interactive
• Assignee: Ning
• Reporter: Ning
• Created: 09/May/22 18:24
• Updated: 31/May/22 18:03
• Resolved: 31/May/22 18:03

Description

Provide the capability for the user to explicitly provision a cluster.

Current implementation provisions each cluster at the location specified by GoogleCloudOptions using 3 worker nodes. There is no explicit API to configure the number or shape of workers.

We could use the WorkerOptions to allow customers to explicitly provision a cluster and expose an explicit API (with UX in notebook extension) for customers to change the size of a cluster connected with their notebook (until we have an auto scaling solution with Dataproc for Flink).

The API looks like this when configuring the workers for a dataproc cluster when creating it:

An example request setting the masterConfig and workerConfig:

Issue Links

• links to: GitHub Pull Request #17736