[MAPREDUCE-2167] Faster directory traversal for raid node

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.22.0
• Component/s: contrib/raid
• Hadoop Flags: Reviewed
• Assignee: Ramkumar Vadali
• Reporter: Ramkumar Vadali
• Created: 29/Oct/10 19:00
• Updated: 12/Dec/11 06:18
• Resolved: 12/Nov/10 00:02

Description

The RaidNode currently iterates over the directory structure to figure out which files to RAID. With millions of files, this can take a long time - especially if some files are already RAIDed and the RaidNode needs to look at parity files / parity file HARs to determine if the file needs to be RAIDed.

The directory traversal is encapsulated inside the class DirectoryTraversal, which examines one file at a time, using the caller's thread.

My proposal is to make this multi-threaded as follows:

• use a pool of threads inside DirectoryTraversal
• The caller's thread is used to retrieve directories, and each new directory is assigned to a thread in the pool. The worker thread examines all the files the directory.
• If there sub-directories, those are added back as workitems to the pool.

Comments?