In the past we have had cases of special agents (e.g. nodes with gpu, nodes with non-default region) where we've added one off opt-in behavior for getting offers from those nodes. This opt-in behavior for getting resources from these agents has been based on framework capabilities (e.g. GPU_RESOURCES, REGION_AWARE), however a generalized mechanism based on attributes would be useful for future cases.
An "opt-in" agent attribute means that the framework has to opt-in its role(s) to specific opt-in attributes in order to obtain resources from agents that have the attribute.
For example, we have a cluster with 5 gpu nodes and want to ensure that these are used for only gpu workloads. The operator would tag these machines with a "gpu_workload" attribute and mark this attribute as opt-in. Now, only those roles that frameworks opt-in to the "gpu_workload" attribute will receive offers for these agents.