[TEZ-4067] Tez Speculation decision is calculated on each update by the dispatcher

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 08/May/19 19:24
• Updated: 09/Dec/19 22:05
• Resolved: 26/Nov/19 21:18

Description

LegacySpeculator is an object field in VertexImpl. Therefore, all events are handled synchronously by the caller (dispatcher). This implies the following:

1. the dispatcher spends long time executing updateStatus as it needs to check the runtime estimation of the tezAttempts within the vertex.
2. the speculator is per stage: lunching a speculation may not the optimum decision. Ideally, based on resources, speculated tasks should be the ones with slowest progress.
3. the time between speculation is skewed because there is a big delay for the dispatcher to complete a full cycle. Also, speculation will be more aggressive compared to MR because MR waits for "soonest.retry.after.speculate" whenever a task is speculated. On the other hand, Tez speculates more tasks as it processes stages in parallel.

 

Issue Links

• is related to: TEZ-1897 Create a concurrent version of AsyncDispatcher
• is required by: TEZ-4106 Add Exponential Smooth RuntimeEstimator to the speculator
• supercedes: TEZ-3934 LegacySpeculator sometime issues wrong number of speculative attempts