[METRON-416] Provide the ability to store mergeable data structures for summarizing data on-line

• Type: Improvement
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.2.1BETA
• Assignee: Casey Stella
• Reporter: Casey Stella
• Created: 13/Sep/16 18:16
• Updated: 27/Sep/16 15:08
• Resolved: 27/Sep/16 14:10

Description

With the currently worked on profiler, we it will be advantageous for us to be able to store more than just numeric data as a profile. In particular, adding some common data structures that can be merged will allow us to store data at a fixed tick-rate, but merge the results from hbase across multiple ticks.

As a first cut, the following structures should be supported:

• An implementation of the STATS stellar functions which is backed by an online and mergeable class using distributional sketches for querying distribution.
• An implementation of bloom filters so that probabalistic existence queries can be made

Issue Links

• links to: GitHub Pull Request #250