Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
To better support long running jobs and services, we need to extend YARN to support other resources, such as disk, IP, port. Current resource types is not flexible enough to make this work because it only supports COUNTABLE type which is single value.
Propose to extend resource types by adding two more general types, such as SET, MULTIDIMENSIONAL (naming TBD). With schema like
SET: a set of values
["10.100.0.1", "10.100.0.2"] ["9981", "9982", "9983"]
MULTIDIMENSIONAL: a set of values, each value can be a resource instance with multiple values.
[ disk1 : { attributes: { "type" : "SATA", "index" : 1 }, "size" : "500gb", "iops" : "1000" },
disk2 : { attributes: { "type" : "SSD", "index" : 2 }, "size" : "100gb", "iops" : "1000" } ]
this way, we could support better resource management and isolations. The idea is to make this as general as possible so we can easily support some other complex resources.