[HDFS-8873] Allow the directoryScanner to be rate-limited

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha1
• Component/s: datanode
• Labels: 2.7.2-candidate
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Templeton
• Reporter: Nathan Roberts
• Created: 07/Aug/15 16:39
• Updated: 18/Aug/17 11:23
• Resolved: 27/Apr/17 22:06

Description

The new 2-level directory layout can make directory scans expensive in terms of disk seeks (see HDFS-8791) for details.

It would be good if the directoryScanner() had a configurable duty cycle that would reduce its impact on disk performance (much like the approach in HDFS-8617).

Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common case of all inodes in cache but no directory blocks cached, 64K seeks are required for full directory listing which translates to 655 seconds)

Issue Links

• breaks: HDFS-12319 DirectoryScanner will throw IllegalStateException when Multiple BP's are present
• breaks: HDFS-9174 Fix findbugs warnings in FSOutputSummer.tracer and DirectoryScanner$ReportCompiler.currentThread
• breaks: HDFS-11707 TestDirectoryScanner#testThrottling fails on OSX
• is duplicated by: HDFS-8989 Allow rate-limiting the DirectoryScanner via time-slicing
• is required by: HDFS-9176 TestDirectoryScanner#testThrottling often fails.
• relates to: HDFS-8791 block ID-based DN storage layout can be very slow for datanode on ext4