[REEF-1357] Allow different caching levels for caching in IInputPartition

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: IMRU, REEF.NET
• Labels: FT
• Assignee: Andrew Chung
• Reporter: Andrew Chung
• Created: 25/Apr/16 20:08
• Updated: 16/Nov/16 20:37

Description

Caching in IInputPartition is currently entirely dependent on the implementation of the IInputPartition. We should modify/expand IInputPartition such that the data loading component is fully separate from the caching component. An idea is to instead provide only a single concrete IInputPartition implementation (while deprecating the old implementations) and use the composition model, complete with an IDataLoader which fetches data back in a Stream, and an IDataCache, which allows caching at granular levels.

Issue Links

• blocks: REEF-1397 Revise FileSystemInputPartition and RandomInputPartition with DataCache and DataMover
• blocks: REEF-1405 Cache the data in IMRU Context layer
• is blocked by: REEF-1389 Add wrapper interface for DirectoryInfo and FileInfo to facilitate testing for REEF-1357
• is related to: REEF-1223 IMRU Fault Tolerance - restart failed evaluators
• is related to: REEF-1365 Define semantics of IFileDeSerializer, CopyToLocal, and their usage in FileSystemInputPartition
• is related to: REEF-1339 Adding IInputPartition.Cache() for data download and cache
• links to: Prototype 1
• links to: Prototype 2