[CLIMATE-522] Implement abstraction layer in climate for accessing other datastores

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: data sources
• Assignee: Maziyar Boustani
• Reporter: Lewis John McGibbney
• Created: 28/Sep/14 02:56
• Updated: 28/Mar/18 13:58

Description

Before long we will wish to support more than what we currently can.
There is a service at
https://github.com/apache/climate/blob/master/rcmet/src/main/python/rcmes/storage/rcmed.py#L26
which provides us with a list of parameters from which we obtain data, however it is only natural that the next step is to provide other data sources over and above this one... baring in mind that a GET request of the current URL above currently gives me a temp unavailable reponse!

For this issue I would propose that we consider an existing framework for providing access to climate data. After some primative research I've compiled the following list

• https://github.com/jpvanhal/siilo - provides access to Local Filesystem, Apache Libcloud and Amazon S3. Libcloud provides access to a plethora of cloud storage services.
• https://github.com/okfn/ofs - S3, Google Storage, Eucalytus, Archive.org, Filesystem (via pairtree), 'REST' Store and Riak (buggy)
• https://github.com/apache/gora/ - Access to HBase, Accumulo, Cassandra, HDFS, Local Filesystem, DynamoDB, Solr, MongoDB and other stores coming soon.

The main motivation here is to open up climate for accesing more data. I am personally VERY interested in pursuing the ticket for GORA-342 which we could hopefully bake the existing rcmed code into.