[DRILL-5028] Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS.

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: Functions - Drill
• Assignee: Unassigned
• Reporter: Account Not Used
• Created: 10/Nov/16 06:45
• Updated: 04/Nov/19 12:37

Description

We have a Drill cluster with 20+ Nodes and we store all history profiles in hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower while serving more queries.

Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching the latest 100 history profiles by default, I guess this operation blocks the page loading (Millions small files can be stored in the basePath), maybe we can try some other ways to reach the same goal.

Issue Links

• relates to: DRILL-5270 Improve loading of profiles listing in the WebUI