Details
-
New Feature
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Client-Side input
The user of the IMRU API will have to provide:
- A REEF Runtime Configuration. This will be used to submit the IMRU Driver.
- Configurations for the IMRU components: Map, Reduce and Update function as well as the needed Codec configurations.
- A Configuration that allows us to instantiate an instance of the (yet to be defined) IInputProviderDriver API.
- A Configuration for the yet to instantiate the (yet to be defined) IOutputProviderDriver
Driver side
Constructor
- Use IInputProviderDriver to get partition information:
- Number of partitions
- Partition ID
- Partition locations (not immediately, but we will want this in the future)
- Use that information to configure Group Communications with:
- Data Broadcast and Reduce
- Control Broadcast (this is to indicate to the Mappers when to close)
OnNext(DriverStarted)
- Request the Evaluators as per the data provider's information
- Set a timeout for this request to be satisfied. If it is not within that time, exit the Driver.
OnNext(AllocatedEvaluator)
- For the UpdateFunction:
- Assemble the Service Configurations for group communication and data output
- Submit the UpdateTask
- For the MapFunction
- Assemble the Service Configurations for group communication and data input
- Submit the MapTask
OnNext(CompletedTask) and OnNext(CompletedEvaluator)
- Keep track of all Tasks and make sure that they all exit cleanly
- Set a timer when the first task completes and fail when not all tasks complete by the end of the timer.
OnNext(FailedTask) and OnNext(FailedEvaluator)
- Fail the Driver (for now)
UpdateTask
Constructor
- Establish and validate data output
- Establish and validate group communications
Call() main loop
- Use the IUpdateFunction instance to determine whether there is a next iteration
- If yes:
- Send control message to the mappers, followed by the data message
- If no:
- Send the ending control message to the mappers
- If the IUpdateFunction provided output, send it.
MapTask
Constructor
- Establish and validate data input
- Establish and validate group communications
Call() main loop
- Wait for control message.
- If one more iteration, call the IMapFunction with the data sent on the data broadcast.
- If the computation is to end, exit the loop.