[REEF-554] Update IMRU APIs to enable writing of REEF IMRU driver and client

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13
• Component/s: REEF.NET

Environment

C#

• Assignee: Dhruv Mahajan
• Reporter: Dhruv Mahajan
• Created: 03/Aug/15 22:32
• Updated: 07/Aug/15 15:56
• Resolved: 07/Aug/15 15:56

Description

The aim of this jira is to modify IMRU APIs to enable implementation of REEF IMRU driver and client on top. Specifically,

a) APIs will be modified. Specifically,

• Instead of a single combined configuration supplied to the driver, it will get configuration separately for different components like Map function, Update function, Codecs, Reduce function. This is done because a) On the user side we in any case take them as separate ones, b) On driver not all evaluators need all the configurations, for example evaluator being assigned the Map function does not need Update function configuration and c) Merging all configurations together has issues of potential conflicts: for example TMapInput and TMapOutput using similar base codec for example VectorCodec.

• Introduce extra user specifications corresponding to PipelineDataConverters for TMapInput and TMapOutput in IMRUJobDefinition, Partitioned Dataset and number of mappers. The number of mappers is redundant since Partitioned Dataset will also have the information, but will be needed in Group Communication.

b) MapperCount example will be modified to be compatible with new API
c) InProcessIMRU will be modified to be compatible with the current API