Hi Tao Jie
Trying to understand the problem here. SO you mentioned that few applications were preempted and became slow, correct?. This can happen only if the resource usage went across the desired configured limit for that Queue. And there were demands (some applications waiting to get containers) to an under-serving queue. This would have resulted in preemption.
In preemption, we will be considering the last submitted applications as first candidate to preempt (within each application, will consider the lowest priority containers) within queue. Few tuning could be done to avoid some sudden trigger of preemption such as "dead-zone" etc. Pls check the preemption related configurations for same.
Now coming to the suggestion, I fee its kind of a hack over preemption. Preemption policy now chooses the apps/containers based on above explained criteria. And if we have more apps labelled with "do-not-preempt", I feel it may not give a fair preemption across applications in a queue. I am not very sure whether we need to give some extra power to a queue/user to save some v.imp apps. But needs to see how far it will help in reality.