[REEF-429] Implement IMRU on Group Communications

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF.NET
• Assignee: Dhruv Mahajan
• Reporter: Markus Weimer
• Created: 29/Jun/15 21:12
• Updated: 15/Sep/15 18:19

Description

Client-Side input

The user of the IMRU API will have to provide:

• A REEF Runtime Configuration. This will be used to submit the IMRU Driver.
• Configurations for the IMRU components: Map, Reduce and Update function as well as the needed Codec configurations.
• A Configuration that allows us to instantiate an instance of the (yet to be defined) IInputProviderDriver API.
• A Configuration for the yet to instantiate the (yet to be defined) IOutputProviderDriver

Driver side

Constructor

• Use IInputProviderDriver to get partition information:
• Number of partitions
• Partition ID
• Partition locations (not immediately, but we will want this in the future)
• Use that information to configure Group Communications with:
• Data Broadcast and Reduce
• Control Broadcast (this is to indicate to the Mappers when to close)

OnNext(DriverStarted)

• Request the Evaluators as per the data provider's information
• Set a timeout for this request to be satisfied. If it is not within that time, exit the Driver.

OnNext(AllocatedEvaluator)

• For the UpdateFunction:
• Assemble the Service Configurations for group communication and data output
• Submit the UpdateTask
• For the MapFunction
• Assemble the Service Configurations for group communication and data input
• Submit the MapTask

OnNext(CompletedTask) and OnNext(CompletedEvaluator)

• Keep track of all Tasks and make sure that they all exit cleanly
• Set a timer when the first task completes and fail when not all tasks complete by the end of the timer.

OnNext(FailedTask) and OnNext(FailedEvaluator)

• Fail the Driver (for now)

UpdateTask

Constructor

• Establish and validate data output
• Establish and validate group communications

Call() main loop

• Use the IUpdateFunction instance to determine whether there is a next iteration
• If yes:
• Send control message to the mappers, followed by the data message
• If no:
• Send the ending control message to the mappers
• If the IUpdateFunction provided output, send it.

MapTask

Constructor

• Establish and validate data input
• Establish and validate group communications

Call() main loop

• Wait for control message.
• If one more iteration, call the IMapFunction with the data sent on the data broadcast.
• If the computation is to end, exit the loop.

Issue Links

• is part of: REEF-424 Add Iterative Map-Reduce-Update
• relates to: REEF-707 Cannnot pass dlls to driver from IMRU Job Definition

Sub-Tasks

• Introduce timeout for Evaluator request to be satisfied in IMRU: Open
• IIMRUClient design needs improvement: Open