[AIRAVATA-3322] Add paging to experiment statistics

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Django Portal, Registry API
• Assignee: Marcus Christie
• Reporter: Marcus Christie
• Created: 16/Apr/20 21:34
• Updated: 07/Jun/22 14:32
• Resolved: 07/Jun/22 14:32

Description

The getExperimentStatistics API method works well for small numbers of experiments but doesn't scale well to thousands of experiments, which some popular gateways like seagrid.org seeing this level of activity. For example, for SEAGrid we had to up the Apache/Django timeout to 5 minutes to load statistics for a few months for one compute resource, and even then it didn't always load.

I added slow query logging in the MariaDB but that turned up nothing so I don't believe the queries need to be tuned. I think the main culprit is that thousands of ExperimentSummary instances are getting populated and returned by the method. It would be better to page in experiments and only return the first, say, 50 experiments in each status category.

Currently the ExperimentSummaries are loaded for each ExperimentState and they are counted. Instead, we'd need two queries per state, one to load the count as a scalar and another to load a page's worth of ExperimentSummaries.

To preserve backwards compatibility, it might be best to just make a new method and then in time phase out the old one.