[HDFS-13692] StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0, 3.1.1, 3.0.4
• Target Version/s: 3.2.0, 3.1.1
• Hadoop Flags: Reviewed
• Assignee: Bharat Viswanadham
• Reporter: Yiqun Lin
• Created: 21/Jun/18 06:23
• Updated: 03/Jul/19 04:41
• Resolved: 22/Jun/18 02:57

Description

StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet.  In StorageInfoDefragmenter#scanAndCompactStorages, it prints for all the StorageInfo under each DN. If there are 1k nodes in cluster, and each node has 10 data dir configured, it will print 10k lines every compact interval time (10 mins). The log looks large, we could switch log level from INFO to DEBUG in StorageInfoDefragmenter#scanAndCompactStorages.

2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 : 0.8222276264591439
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 : 0.9330040998881849
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb : 0.9314626719970249
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd : 0.9309580852251582
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 : 0.8938870614035088
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa : 0.8963103205353998
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 : 0.8950508004926109
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 : 0.8947356866877415

Issue Links

• duplicates: HDFS-14628 Improve information on scanAndCompactStorages in BlockManager and lower log level