Ignore the previous comment (which may be due to a bug we are trying to diagnose separately). i would like to stick to the initial report - deficit based scheduling causes pending jobs to rise up in priority (effectively). especially when such jobs are large - they tend to hog the cluster once they can be scheduled.
i guess this does beg the question of why a large deficit was accumulated in the first place (and that maybe due to a bug) - but this does seem to call for some solution in any case.
one of the things that i think u had mentioned would make sense - instead of giving all incoming slots to a task with a large deficit - give it a large enough fraction so that it's 'catching up'. one way could be to assign additional weight multiplicator that's proportional to the deficit. this should leave some slots on an ongoing basis to new jobs without a lot of deficit. thoughts?