I have a job which after a while in one of its stages grinds to a halt, from processing around 300k tasks in 15 minutes to less than 1000 in the next hour. The driver ends up using 100% CPU on a single core (out of 4) and the executors start failing to receive heartbeat responses, tasks are not scheduled and results trickle in.
For this stage the max scheduler delay is 15 minutes, and the 75% percentile is 4ms.
It appears that TaskScheulderImpl does most of its work whilst holding the global synchronised lock for the class, this synchronised lock is shared between at least,
This looks to severely limit the latency and throughput of the scheduler, and casuses my job to straight up fail due to taking too long.