[HDFS-14187] Make warning message more clear when there are not enough data nodes for EC write

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: erasure-coding
• Assignee: Kitti Nanasi
• Reporter: Kitti Nanasi
• Created: 07/Jan/19 15:22
• Updated: 03/Oct/19 23:18
• Resolved: 01/Feb/19 00:08

Description

When setting an erasure coding policy for which there are not enough racks or data nodes, write will fail with the following message:

[root@oks-upgrade6727-1 ~]# sudo -u systest hdfs dfs -mkdir /user/systest/testdir
[root@oks-upgrade6727-1 ~]# sudo -u hdfs hdfs ec -setPolicy -path /user/systest/testdir
Set default erasure coding policy on /user/systest/testdir
[root@oks-upgrade6727-1 ~]# sudo -u systest hdfs dfs -put /tmp/file1 /user/systest/testdir
18/11/12 05:41:26 WARN hdfs.DFSOutputStream: Cannot allocate parity block(index=3, policy=RS-3-2-1024k). Not enough datanodes? Exclude nodes=[]
18/11/12 05:41:26 WARN hdfs.DFSOutputStream: Cannot allocate parity block(index=4, policy=RS-3-2-1024k). Not enough datanodes? Exclude nodes=[]
18/11/12 05:41:26 WARN hdfs.DFSOutputStream: Block group <1> failed to write 2 blocks. It's at high risk of losing data.

I suggest to log a more descriptive message suggesting to use hdfs ec -verifyCluster command to verify the cluster setup against the ec policies.