Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.3.0
-
None
Description
Speculation currently has four different configuration options according to the docs:
- spark.speculation to turn it on or off
- .interval - how often it'll check
- .multiplier - how many times slower than average a task must be
- .quantile - fraction of tasks that have to be completed before speculation starts
What I'd love to see is a feature that allows me to set a minimum average duration. We are using a Jobserver to submit varying queries to Spark. Some of those have stages with tasks of an average length of ~20ms (yes, not idea but it happens). So with a multiplier of 2 a speculative task starts after 40ms. This happens quite a lot and in our use-case it doesn't make sense. We don't consider 40ms tasks "stragglers".
So it would make sense to add a parameter to only start speculation when the average task time is greater than X seconds/minutes/millis.