[HDFS-4412] Support HDFS IO throttling

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Zhenxiao Luo
• Created: 16/Jan/13 19:50
• Updated: 27/Aug/15 12:27

Description

When an applications upload/download files from/to HDFS clusters, it would be nice if the IO could be throttled so that they won't go beyond the specified maximum bandwidth.

Two options to implement this IO throttling:

#1. IO Throttling happens at the FSDataInputStream and FSDataOutputStream level.

Add an IO Throttler to FSDataInputStream/FSDataOutputStram, and whenever an read/write happens, throttle it first(if throttler is set), then do the actual read/write.

We may need to add new FileSystem apis to take an IO throttler as input parameter.

#2. IO Throttling happens at the application level.

Instead of changing the FSDataInputStream/FSDataOutputStream, all IO throttling is done at the application level.

In this approach, FileSystem api remains unchanged.

Either case, an IO throttler interface is needed, which has a:
public void throttle(long numOfBytes);

The current DataTransferThrottler could be an implementation of this IO throttler interface.

Issue Links

• is related to: HADOOP-9194 RPC Support for QoS