[OAK-891] Use DirectMemory as Level 2/Offheap cache

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8
• Fix Version/s: 0.16
• Component/s: core
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 09/Jul/13 08:58
• Updated: 10/Feb/14 13:51
• Resolved: 05/Feb/14 14:39

Description

To make optimum use of memory allocated to the various caches created in Oak it would be better to use Apache DirectMemory [1] as the level 2 (L2) cache. Further using Kryo[2] Serializer would enable saving of serialized objects in more compact form

• Storing cache objects in serialized form provides a more compact storage
• Deserializtion cost is low
• Using Off heap memory would be helpful compared to on heap memory
• L2 cache can be added as an optional feature. Oak Core can still be
  used in absence of L2 cache

Implementation details

• Feature Branch
• Diff
• Test cases to be added

Implementation notes

• Adds DirectMemory and Kryo library as optional dependencies
• Decorates the Guava Cache to move the evicted objects to L2 cache
• Using L2 cache is completely optional
• Kryo library is used to provide fast and compact serialization.
• Uses custom serializers for various cache instances to provide a more compact storage
• Instead of using the key as is (i.e. path in MongoMK) it stores the hash of the path to conserve memory

Initial discussion at http://markmail.org/thread/xief4l25m7tnx52c

[1] http://directmemory.apache.org/
[2] https://code.google.com/p/kryo/