[GEARPUMP-85] Shared Data Types

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Component/s: restapi
• Assignee: Unassigned
• Reporter: Kam Kasravi
• Created: 21/Apr/16 17:08
• Updated: 17/May/16 07:43

Description

GearPump Shared Data Types

Definition

A shared data type is a type that crosses the REST layer either from the client (eg POST) or to the client (eg GET).

Requirements

1. Shared data types should be easily demarked. We should have a way of easily identifying these data types which is automated. That is - a programmatic way of finding these data types reliably that requires no separate list or manual maintenance which could become outdated.
2. The REST API should have documentation auto-generated. This includes the endpoints and the shared data types. It should be part of the build process.
3. All shared data types should be JSON friendly. That is no data type should require special serde (serialization/deserialization) methods on client or server to handle unserializable members.
4. There should be compilation checks that prevent type related errors. That is changing a shared data type on either the client or server will result in a compilation error on the flip side.
5. Shared data types should not need to be repackaged. Current types should be able to remain as is, new shared data types should be able to be defined where appropriate.
6. Annotations or other constructs should be minimal. Given the constraint that shared data types can be defined anywhere, there needs to be a way to identify them. Naming conventions are not a possibility due to no naming convention has been followed to date. If annotations are used they should be minimal.
7. Shared data types should provide a [json schema (v4)](http://json-schema.org/documentation.html) or equivalent that allows tooling to transpile these shared data types to potentially other languages. 3rd party clients may require a schema to develop any reasonable client-side functionality within reasonable time frames.

Use Cases

GearPump provides a REST API that is used to expose a wide variety of functionality both for the user of GearPump and the administrator. The REST API leverages JSON in many HTTP GET, POST, PUT, DELETE operations. These JSON structures are used in javascript as javascript objects, within the REST server and within the master, worker and executor actors. They permeate all environments. Given their scope and prevalence we should have a set of guidelines on how these data types should be created, changed and transported. This will minimize a variety of problems.

Current usage

Currently shared data types as defined above are found across a large number of packages on the server and inline with other classes, objects and other scala definitions. There is no devoted package or equivalent construct to isolate these data types. They are also not annotated or otherwise identified within the code base. That is - there is no way to reliably gather these types other than a tedious review of the services classes and their route paths. Additionally, within the client layer there is no typing of these shared data types within javascript. For GET operations these data types are reified into untyped javascript object literals, where field naming is preserved but type naming is not. In other words a scala data type of `case class MasterData(masterDescription: MasterSummary)` is not identified by that name in the client code. There is also no tooling in place to auto-generate documentation for these shared data types or the REST API in general. Any 3rd party wishing to build their own client would need to scour the code and attempt to cobble together the REST endpoints and their related data. This would be untenable for most 3rd parties and represents a significant obstacle in promoting GearPump's many other strengths in a very competitive landscape.

Positives

There are some things that are done on the server with these shared data types that are positive.

1. upickle is used to read and write all shared data types on the server. (Unfortunately upickle is not used on the client).
2. There is minimal specialized type conversions. Upickle provides a way to take a type that cannot normally be serialized and allows one to provide a customized JSON mapping. For example Actor is not a serializable data type, if one were to serialize an Actor using upickle they would need to provide a customized serializer/deserializer for it. There are few places (if any) where we do this.
3. All data types are case classes.

Suggested Design approaches

Design solutions should attempt to leverage best-in-class tooling - for example for REST related documentation. Any design solution should not tax the current work flow with a heavy weight process related to development on either the client or server side.