[HDFS-7276] Limit the number of byte arrays used by DFSOutputStream

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: hdfs-client
• Hadoop Flags: Reviewed

Release Note

The following configuration properties are added.

- dfs.client.write.byte-array-manager.enabled:
for enabling/disabling byte array manger. Default is false.

- dfs.client.write.byte-array-manager.count-threshold:
The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128.

- dfs.client.write.byte-array-manager.count-limit:
The maximum number of arrays allowed for each array length. Default is 2048.

- dfs.client.write.byte-array-manager.count-reset-time-period-ms:
The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.

• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 22/Oct/14 02:24
• Updated: 01/Dec/14 03:08
• Resolved: 01/Nov/14 18:36

Description

When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding packets could be large. The byte arrays created by those packets could occupy a lot of memory.

Issue Links

• breaks: HDFS-7358 Clients may get stuck waiting when using ByteArrayManager
• relates to: HDFS-7321 Add a mechanism to clean up the free queue in ByteArrayManager
• relates to: HDFS-7260 Make DFSOutputStream.MAX_PACKETS configurable
• relates to: HDFS-7308 DFSClient write packet size may > 64kB