[HADOOP-1189] Still seeing some unexpected 'No space left on device' exceptions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.2
• Fix Version/s: 0.13.0
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 31/Mar/07 03:19
• Updated: 08/Jul/09 16:42
• Resolved: 11/Apr/07 19:04

Description

One of the datanodes has one full partition (disk) out of four. Expected behaviour is that datanode should skip this partition and use only the other three. HADOOP-990 fixed some bugs related to this. It seems to work ok but some exceptions are still seeping through. In one case there 33 of these out 1200+ blocks written to this node. Not sure what caused this. I will submit a patch to the prints a more useful message throw the original exception.

Two unlikely reasons I can think of are 2% reserve space (8GB in this case) is not enough or client some how still says block size is zero in some cases. Better error message should help here.

If you see small number of these exceptions compared to number of blocks written, for now you don't need change anything.