[GOBBLIN-2085] Increase `startToCloseTimeout` for `ExecuteGobblinWorkflow` activities

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-core
• Assignee: Abhishek Tiwari
• Reporter: Kip Kohn
• Created: 12/Jun/24 02:44
• Updated: 12/Jun/24 18:55

Description

the currently hard-coded `startToCloseTimeout` values are too short. requirements that have come to light w/ operational experience:

• `ProcessWorkUnits` must support extractors utilizing little parallelism (such as those reading from a DB) that take a very long time, even upwards of hours
• `CommitActivity` runs may have to handle O(10k) or more task state files, which may take a long time to open and read, esp. when the `FileSystem` is under heavy load
• `GenerateWorkUnits` must work with sources that may be quite vast, yet w/ limited recourse to parallelism, such as a massive source iceberg

ultimately these and other temporal config values would ideally be configurable, and that will come soon. for now, just bump to values large enough to resolve site-up issues.