[HDFS-15461] TestDFSClientRetries#testGetFileChecksum fails intermittently

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.2, 3.3.1, 3.4.0, 3.2.3
• Component/s: dfsclient, test
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 08/Jul/20 17:40
• Updated: 30/Oct/20 09:22
• Resolved: 27/Oct/20 08:14

Description

TestDFSClientRetries.testGetFileChecksum fails intermittently on hadoop trunk

[INFO] Running org.apache.hadoop.hdfs.TestGetFileChecksum
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.491 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestGetFileChecksum
[ERROR] testGetFileChecksum(org.apache.hadoop.hdfs.TestGetFileChecksum)  Time elapsed: 4.248 s  <<< ERROR!
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:52468,DS-e35b6720-8ac2-4e5e-98df-306985da6924,DISK], DatanodeInfoWithStorage[127.0.0.1:52472,DS-91ec34d5-3f0a-494e-aed6-b01fa0131d8a,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:52472,DS-91ec34d5-3f0a-494e-aed6-b01fa0131d8a,DISK], DatanodeInfoWithStorage[127.0.0.1:52468,DS-e35b6720-8ac2-4e5e-98df-306985da6924,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)
	at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)
	at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)
	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)
	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:719)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   TestGetFileChecksum.testGetFileChecksum » IO Failed to replace a bad datanode ...
[INFO]
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[ERROR] There are test failures.

Issue Links

• is related to: HDFS-13833 Improve BlockPlacementPolicyDefault's consider load logic
• is superceded by: HDFS-15659 Set dfs.namenode.redundancy.considerLoad to false in MiniDFSCluster
• relates to: HDFS-4723 Occasional failure in TestDFSClientRetries#testGetFileChecksum because the number of available xcievers is set too low
• supercedes: HDFS-4723 Occasional failure in TestDFSClientRetries#testGetFileChecksum because the number of available xcievers is set too low
• links to: GitHub Pull Request #2404