[ATLAS-2915] AWS Kinesis Stream Typedef for Atlas

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Barbara Eckman
• Reporter: Barbara Eckman
• Created: 08/Oct/18 15:33
• Updated: 20/Dec/19 18:07

Description

Currently the base types in Atlas do not include AWS Kinesis Stream objects. It would be nice to add a typedef for a kinesis stream, inheriting from DataSet.  Attributes would include:

• streamType string, eg ""Single Region Stream".
• awsRegion string: the AWS region in which the kinesis stream endpoint is deployed
• shardCount int:  number of shards (uniquely identified sequence of data records) in the stream
• streamEnvironment enum.  Valid values are "unknown", "production", "staging", "QA" and "development"
• containsPII boolean: does this stream's data contain Personally Identifiable Information?
• aggregationFormat enum. Indicates if/how the records are aggregated within a single kinesis record. Valid values are "none" or "kpl".
• contentType enum: serialization format used by the producer of the stream.  Valid values are "unknown", "avro", "bson", "csv", "json", "key-value", "kryo", "protobuf", "raw" [ie no consistent schema], "sdp" [confluent-style avro with envelope that specifies schema id surrounding the payload], "thrift", "tlv", "xml", "other".
• schemaURL string: A URL to the data schema used by the producer, to facilitate consumption.
• schemas: array of schema objects associated with the kinesis stream. Typically avro schemas but could be JSON schema, etc.