Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.4.0
-
None
Description
Our current ESGF loader currently has several limitations:
- Only one dataset can be downloaded / processed at a time
- Datasets must be specified by dataset_id, which requires users to perform manual searches to obtain
- Lack of customization options in search
Since our original loader was designed, ESGF's REST API (https://www.earthsystemcog.org/projects/cog/esgf_search_restful_api) has come a long way in terms of ease of use and extensibility. It can be used to automatically generate a shell script which can download all of the data. This can be easily done using either our own simple implementation or esgf-pyclient's search API.