[HDFS-14076] NameNodeResourceChecker#isResourceAvailable() should check the linux filesystem inode usage.

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: namenode
• Assignee: Surendra Singh Lilhore
• Reporter: Surendra Singh Lilhore
• Created: 14/Nov/18 06:51
• Updated: 16/Nov/18 15:36

Description

Linux system gives the "No space left on device" exception in two cases.

1. Disk space is not available 
2. Linux inode limit is exceeded

NameNodeResourceChecker currently only checking the disk space, it should check the inode limit also.

We got "No space left on device" exception even disk space was there and roll edit operation is failed. After analysis we found that Inode usage for the system is 100%.

2018-11-10 18:59:37,913 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: starting log segment 796xxxxxx failed for (journal JournalAndStream(mgr=FileJournalManager(root=/opt/xxxxxxxxxx), stream=null))
java.io.FileNotFoundException: /opt/xxxxxxx/edits_inprogress_0000000000007964819 (No space left on device)
 at java.io.RandomAccessFile.open0(Native Method)
 at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
 at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
 at org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.<init>(EditLogFileOutputStream.java:88)