[CRUNCH-256] SequentialFileNamingScheme should cache the # of files in the target directory after the first read

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.8.0
• Assignee: Josh Wills
• Reporter: Josh Wills
• Created: 23/Aug/13 00:10
• Updated: 08/Nov/13 21:24
• Resolved: 24/Aug/13 20:12

Description

After a job finishes running, the post-job hooks rename the files from a temp output directory to the target output directory. When we have lots of files, this move can take a long time, and I traced the performance issue to the fact that SequentialFileNamingScheme does a listStatus() on the output directory for every file that gets moved. If SequentialFileNamingScheme just does this check once and then increments an internal counter, we can significantly decrease the performance overhead involved with the move.