Major Description
This task will be to make the new version of the Block Cache production ready.
The code can be found in:
https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=shortlog;h=refs/heads/0.2.1
The big features for this new code are the following:
1. Variable length cache block sizes. The big need for this is in filtering. Creating huge bits sets on the heap to cache filters created by inbound queries can be hard to manage. It can also production performance problems (GC issues) and can cause OOM errors if too many filters are created. If a filter can be written as a file per segment and cached as a single unit in the block cache, then it will be as performant as on heap bitset but will live off the heap and under the same resource constraints placed on the block cache. So basically when a cluster is over allocated (when it comes to caching filters) the cluster will be less likely to fail and instead will perform more slowly.
2. Block cache size can be dynamically changed.
3. Simpler implementation for the off heap slabs of memory. Which should also make it faster.
Attachments
1.
|
Change Block Cache so that merges do not effect the cache |
|
Closed | Unassigned |
2.
|
Need to add a closing queue for the CacheValue |
|
Closed | Unassigned |
3.
|
Create a file based bitset filter |
|
Closed | Unassigned |
4.
|
Configure block cache size in blur-site |
|
Closed | Unassigned |