[KYLIN-603] Add mem store for seconds data latency

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Backlog
• Assignee: liyang
• Reporter: liyang
• Created: 11/Feb/15 07:59
• Updated: 31/Oct/19 03:14
• Resolved: 31/Oct/19 03:14

Description

For seconds level streaming data latency

• Data is separated into Young and Aged.

• Add mem store to hold young generation micro batch loaded from streaming (e.g. every 15 seconds). Mem store is good for high throughput and frequent access.

• Aged generation is batch loaded from mem store to disk store daily or hourly.

• Data is persisted in two copies, one as Inverted Index, one as Cube, for both young and aged.

• Inverted Index is for analysis at raw record or low aggregation levels (cardinality > 10K), optimized for parallel random read.

• Cube is for analysis at high aggregation levels (cardinality < 10K), optimized for sequential scan.

• Hybrid bring aged from cube and young from inverted index when young cube is omitted.

Issue Links

• relates to: KYLIN-599 Near real-time support