[REEF-1339] Adding IInputPartition.Cache() for data download and cache

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15
• Labels: FT
• Assignee: Andrew Chung
• Reporter: Julia Wang
• Created: 14/Apr/16 02:01
• Updated: 28/Apr/16 20:35
• Resolved: 28/Apr/16 20:35

Description

Currently, in FileSystemInputPartition, data downloading is implemented in Initilaize() and called from GetPartitionHandle. It doesn't give client a flexibility to decide when to download data. Besides, if client wants to cache data in advance, they need to call GetPartitionHandle() and iterate the data.

We would like to expose a new API Cache() in IInputPartition which performs data download to RAM, SSD, HDD, etc based on client's configuration.

The method should be called in ContextStartHandler in IMRU scenarios.

Issue Links

• Is contained by: REEF-1223 IMRU Fault Tolerance - restart failed evaluators
• Is contained by: REEF-1224 IMRU Fault Tolerance - Separate Data downloading from Task injection
• is related to: REEF-1365 Define semantics of IFileDeSerializer, CopyToLocal, and their usage in FileSystemInputPartition
• relates to: REEF-1357 Allow different caching levels for caching in IInputPartition
• links to: Link to PR