[GEARPUMP-149] Binding tasks to workers

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Karol Brejna
• Created: 21/May/16 12:11
• Updated: 27/May/16 06:09

Description

Let's assume that we have a cluster with some "specialized" nodes (for example a worker on IoT device that is responsible for controlling some actuators, dedicated workers to achieve data locality, some hardware specific - mem/cpu - tasks, etc.).

There should be a way to "hint" Gearpump to prefer some nodes (workers) over others for certain tasks (executors).

We already have "localities" feature. Developer can add a configuration (when submitting the application) to bind a task with a worker:

gearpump {
  streaming {
    localities {
      app1: {            //This is the application name to be submitted
        workerId: [    //The worker id where to launch Task0_0 and Task0_1
          TaskId(0,0), TaskId(0,1)
        ]
      }
    }
  }
}

The problem is: users have no idea what each Task's task ID is (especially if there are lots of nodes in a DAG).

We could introduce more flexible "binding" mechanism, for example labeling tasks and workers or using some other "awareness" algorithm.

Issue Links

• is part of: GEARPUMP-145 Gearpump on Raspberry Pi