[HDFS-17577] Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk Space and Network Load in Labeled YARN Nodes

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: dfsclient
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: liang yu
• Created: 10/Jul/24 03:37
• Updated: 16/Aug/24 19:33

Description

Description: I am currently using Apache Flink to write files into Hadoop. The Flink application runs on a labeled YARN queue. During operation, it has been observed that the local disks on these labeled nodes get filled up quickly, and the network load is significantly high. This issue arises because Hadoop prioritizes writing files to the local node first, and the number of these labeled nodes is quite limited.

 

Problem: The current behavior leads to inefficient disk space utilization and high network traffic on these few labeled nodes, which could potentially affect the performance and reliability of the application. As shown in the picture, the host I circled have a average net_bytes_sent speed 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk space nearly destroyed the whole cluster. 

 

Implementation: The implementation would involve adding an configuration dfs.client.write.no_local_write to support the CreateFlag.NO_LOCAL_WRITE during the file creation process in Hadoop's file system APIs. This will provide flexibility to applications like Flink running in labeled queues to opt for non-local writes when necessary.

 

Issue Links

• links to: GitHub Pull Request #6935