Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13692

StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0, 3.1.1, 3.0.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet.  In StorageInfoDefragmenter#scanAndCompactStorages, it prints for all the StorageInfo under each DN. If there are 1k nodes in cluster, and each node has 10 data dir configured, it will print 10k lines every compact interval time (10 mins). The log looks large, we could switch log level from INFO to DEBUG in StorageInfoDefragmenter#scanAndCompactStorages.

      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 : 0.8222276264591439
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 : 0.9330040998881849
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb : 0.9314626719970249
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd : 0.9309580852251582
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 : 0.8938870614035088
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa : 0.8963103205353998
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 : 0.8950508004926109
      2018-06-19 10:18:48,849 INFO [StorageInfoMonitor] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 : 0.8947356866877415
      

        Attachments

        1. HDFS-13692.00.patch
          1 kB
          Bharat Viswanadham

          Activity

            People

            • Assignee:
              bharatviswa Bharat Viswanadham
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: