Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I have been working on a Puppet module to manage the configuration of VCL and any xCAT tables associated with it. One issue we have run into and are currently discussing is how to manage schema evolution. The update .sql script that creates a series of stored procedures and runs through several case statements to upgrade the schema is difficult to follow, and does not address any other code or supporting stored object definitions that you may have connected to VCL.
Have you considered Avro + a document store? Avro is an Apache project for seralization / deserialization that includes schema validation and nonbreaking transformations, automatically. Objects and schemata can be expressed in JSON, so it would lend itself to a json document store like Apache CouchDB very simply.
You could have for any given storable object, obj, two databases: objdb and objschemadb. Store schema definitions for object obj in objschemadb, and store obj objects in objdb, including a field indicating the schema used for validation. That way, if the schema changes, obj can be read using its old schema and written using the new schema. Avro provides support for simple schema changes, such as adding fields, changing variable names or changing a datatype to a compatible datatype, like int to long. For more complex schema changes, the transformations necessary would have to be done in code, but you would at least know exactly what schemata the data was transforming from and to.
Avro has libraries for both PHP and Ruby, as well as several other languages, and the JSON object format would be well suited to working with data in dojo + javascript. The system configuration could include a target schemata version, and changes to existing data could be applied either during the configuration change of the target version, or lazily on updates to existing data (reading with the previous schema and then writing to the target schema).
Configuration management could define the target schemata by version or "latest". Schemata of individual objects could be defined by version or "latest" or "any" (to allow lazy updates later, for example). Backwards updates should probably be discouraged or not supported, because making changes like changing a field back from a long to an int could lose information. Another advantage of CouchDB is that you could expose read access to stored objects using its API. It might also be possible to integrate schema validation and evolution into CouchDB's javascript validation using an Avro javascript library like this one