[APEXMALHAR-2130] Scalable windowed storage

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 3.7.0
• Labels: roadmap
• Assignee: David Yan
• Reporter: Bright Chen
• Created: 29/Jun/16 23:30
• Updated: 01/Apr/17 05:51
• Resolved: 12/Mar/17 17:05

Description

This feature is used for supporting windowing.

The storage needs to have the following features:
1. Spillable key value storage (integrate with APEXMALHAR-2026)
2. Upon checkpoint, it saves a snapshot for the entire data set with the checkpointing window id. This should be done incrementally (ManagedState) to avoid wasting space with unchanged data
3. When recovering, it takes the recovery window id and restores to that snapshot
4. When a window is committed, all windows with a lower ID should be purged from the store.
5. It should implement the WindowedStorage and WindowedKeyedStorage interfaces, and because of 2 and 3, we may want to add methods to the WindowedStorage interface so that the implementation of WindowedOperator can notify the storage of checkpointing, recovering and committing of a window.

Issue Links

• is blocked by: APEXMALHAR-2026 Spill-able Datastructures
• relates to: APEXMALHAR-2085 Implement Windowed Operators
• relates to: APEXMALHAR-2271 Special SpillableSetMultimap<K, Window.SessionWindow<K>> handling in ManagedState in SpillableSessionWindowedStorage
• links to: GitHub Pull Request #405
• links to: GitHub Pull Request #407
• links to: GitHub Pull Request #408
• links to: GitHub Pull Request #424

Sub-Tasks

• Implement SpillableByteArrayListMultimapImpl.containsEntry: Closed
• Implement SpillableArrayListImpl.iterator: Closed
• Implement a way to get all keys with or without prefix from SpillableByteMapImpl: Closed
• Implement SpillableByteArrayListMultimapImpl.removeAll(Object key): Closed
• Implement SpillableByteArrayListMultimapImpl.remove(@Nullable Object key, @Nullable Object value): Closed
• Make sure unit tests in WindowedOperatorTest all pass with the spillable data structure: Closed
• Implement SpillableByteLinkedListMultimap<K, V>: Closed
• Implement a Spillable map that takes two keys: Closed
• Optimize WindowedStorage and Spillable data structures for time series: Closed
• Create SpillableSet and SpillableSetMultimap interfaces and implementation: Closed
• Create SpillableSet and SpillableSetMultimap interfaces and implementation: Closed
• Implement another TimeBucketAssigner to work with any time: Closed
• Purge the time buckets from managed state for all time buckets that fall beyond the lateness horizon: Closed