By the change in PR of
SPARK-29154, submitted tasks are scheduled onto executors only if resource profile IDs strictly match. As a result Spark always starts new executors for customized ResourceProfiles.
This limitation makes working with process-local jobs unfriendly. E.g. Task cores has been increased from 1 to 4 in a new stage, and executor has 8 slots, it is expected that 2 new tasks can be run on the existing executor but Spark starts new executors for new ResourceProfile. The behavior is unnecessary.