Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Nemo relies on the executor_json flag (defined in JobConf) specified by the user at the command line for acquiring resources. The current behavior is to first acquire all the specified resources upon job start according to the executor_json, and then start executing the IR DAG. Based on the resource-related execution properties annotated on the vertices, Nemo schedules computations across the acquired resources.
This current separation of resource acquisition (via the command line configuration flag) and the resource placement/usage (via the IR DAG annotations) may actually not be desirable for the following reasons. First, this puts an extra burden on both the developers that write passes, and those that deploy jobs. While working on one interface, they need to also think about what happens with the other interface. Second, the configuration parameter is somewhat unnecessarily coarse-grained, as it is applied DAG-wide. There may be optimizations that benefit from more fine-grained resource acquisition (e.g., per-vertex resource acquisition).
A straightforward solution would be to remove the executor_json flag, and incorporate its functionality into the IR DAG, for example through a new execution property. This allows developers to only worry about the passes they apply, and also enables more fine control over which resources to acquire/use for which computations,.