[DRILL-3901] Performance regression with doing Explain of COUNT(*) over 100K files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Assignee: Aman Sinha
• Reporter: Aman Sinha
• Created: 06/Oct/15 18:29
• Updated: 07/Oct/15 17:08
• Resolved: 07/Oct/15 17:08

Description

We are seeing a performance regression when doing an Explain of SELECT COUNT over 100K files in a flat directory (no subdirectories) on latest master branch compared to a run that was done on Sept 26. Some initial details (I will have more later):

master branch on Sept 26
   No metadata cache: 71.452 secs
   With metadata cache: 15.804 secs

Latest master branch 
   No metadata cache: 110 secs
   With metadata cache: 32 secs

So, both cases show regression.

mehant and I took an initial look at this and it appears we might be doing the directory expansion twice.