[HDFS-16293] Client sleeps and holds 'dataQueue' when DataNodes are congested

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.2, 3.3.1, 3.2.3
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: hdfs-client

Language

Java

• Assignee: Yuanxin Zhu
• Reporter: Yuanxin Zhu
• Created: 01/Nov/21 13:12
• Updated: 04/Jan/22 22:45
• Resolved: 06/Dec/21 02:47

Description

When I open the ECN and use Terasort(500G data,8 DataNodes,76 vcores/DN) for testing, DataNodes are congested(HDFS-8008). The client enters the sleep state after receiving the ACK for many times, but does not release the 'dataQueue'. The ResponseProcessor thread needs the 'dataQueue' to execute 'ackQueue.getFirst()', so the ResponseProcessor will wait for the client to release the 'dataQueue', which is equivalent to that the ResponseProcessor thread also enters sleep, resulting in ACK delay.MapReduce tasks can be delayed by tens of minutes or even hours.

The DataStreamer thread can first execute 'one = dataQueue. getFirst()', release 'dataQueue', and then judge whether to execute 'backOffIfNecessary()' according to 'one.isHeartbeatPacket()'

 

Issue Links

• is caused by: HDFS-8008 Support client-side back off when the datanodes are congested