[SAMZA-2783] Re-factor DirDiffUtil.getDirDiff to avoid repeated calls to areSameFile

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4
• Assignee: Unassigned
• Reporter: Andy Sautins
• Created: 26/May/23 21:24
• Updated: 14/Jun/23 21:33
• Resolved: 14/Jun/23 21:33

Description

While profiling a Samza job it was noticed that, for this given job, ~38% of the time was spent in org.apache.samza.storage.blobstore.util.DirDiffUtil.getDirDiff, with the primary contributor being areSameFile.

 

Looking at the code it has the following comment:

DirDiffUtil.java:271

  // TODO MED shesharm: this compares each file in directory 3 times. Categorize files in one traversal instead.

 

Re-factor DirDiffUtil.getDirDiff to loop through all names once, reducing the number of calls to areSameFile.test.