[FLINK-17472] StreamExecutionEnvironment and ExecutionEnvironment in Yarn mode

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Won't Do
• Affects Version/s: 1.10.0
• Component/s: Client / Job Submission, Deployment / YARN
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: RocMarshal
• Created: 30/Apr/20 06:45
• Updated: 11/Aug/21 12:37
• Resolved: 11/Aug/21 12:37

Description

Expect to have such a mode of submission. Build the task directly in the Environment, and then submit the task in yarn mode. Just like RemoteStreamEnvironment, as long as you specify the parameters of the yarn cluster (host, port) or yarn configuration directory and HADOOP_USER_NAME, you can use the topology built by Env to submit the task .

This submission method is best to minimize the transmission of resources required by yarn to start flink-jobmanager and taskmanagerrunner to ensure that flink can deploy tasks on the yarn cluster as quickly as possible.

The simple demo as shown in  the external link Simple outline of the Yarn-Env API（Per-Job Mode) .the parameter named 'env' containes all the operators about job ,like sources,maps,etc..

 

Issue Links

• relates to: FLINK-13938 Use pre-uploaded libs to accelerate flink submission
• relates to: FLINK-14964 Support configure remote flink jar
• relates to: FLINK-17412 Make the FLINK_YARN_FILES value configurable