Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently we allow users to directly program tasklets. There are some issues though.
1. Users have too much freedom in implementing their tasklets. They can make them big, do a blocking call, communicate with each other, etc. We recommend the tasklets to be small and independent to better cope with frequent preemptions, but have no ways to enforce it.
2. Users have to manually decompose work into multiple tasklets. This means that they have to find the optimal tasklet granularity by themselves. It'd be good if Vortex provides a system support for that.
To address these issues, I wonder whether it'd be good to raise the abstraction level. For example, instead of submitting individual tasklets(Data+VortexFunction), users can submit the whole Iterable<Data>+VortexFunction once.
This partially addresses the first problem, at least with regard to the tasklet size. The second problem can be addressed by Vortex runtime dynamically decomposing the work based on the preemption rate.
I'm open to any suggestions or comments.