Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
When handling fetch failures and deciding whether the upstream task should be re-run, we should consider the duration of the upstream task that generated the data trying to be fetched. If the upstream task ran for a long time then we may want to retry a bit harder before deciding to re-run. If the upstream task executed in a few seconds then we should probably re-run the upstream task more aggressively since that may be cheaper than multiple retries that timeout.