[FLINK-35332] Manually setting rest.bind-address does not work for Flink running on Hadoop Yarn dual network environment

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.20.0, 1.19.1, 2.0-preview
• Fix Version/s: 2.0.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Yao Zhang
• Reporter: Yao Zhang
• Created: 13/May/24 07:47
• Updated: 06/Nov/24 12:25
• Resolved: 06/Nov/24 12:24

Description

Given the Hadoop Yarn cluster with dual networks:

• 192.168.x.x: For data transfer. Speed: 10Gbps.
• 10.x.x.x: For management only. Speed: 1Gbps.

 

A client outside the Hadoop Yarn cluster is configured, with management network only(10.x.x.x) and data transfer high speed network not accessible. To reproduce, we sumbit a Flink job from this client(Batch word count for example), the job can be successfully submitted but the result cannot be retrieved, with the exception: Connection refused: {jobmanager_hostname}:{jm_port}. The root cause is the job manager rest address is bind to its actual address (192.168.x.x) rather than 0.0.0.0. Manually setting rest.bind-address does not work.

 

One of the changes in Flink-24474 in YarnEntrypointUtils overwrites RestOptions.BIND_ADDRESS to the node's actual address. This change should be reverted.

Issue Links

• is caused by: FLINK-24474 Standalone clusters should bind to localhost by default
• is duplicated by: FLINK-35747 customer ‘rest.bind-address' config overwrite by code
• links to: GitHub Pull Request #24775