[FLINK-35232] Support for retry settings on GCS connector

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.20.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Oleksandr Nitavskyi
• Reporter: Vikas M
• Created: 24/Apr/24 19:06
• Updated: 09/May/24 01:21
• Resolved: 09/May/24 01:21

Description

https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to specify transport options in GCS connector. While setting the params enabled here reduced read timeouts, we still see 503 errors leading to Flink job restarts.

Thus, in this ticket, we want to specify additional retry settings as noted in https://cloud.google.com/storage/docs/retry-strategy#customize-retries.

We need these methods available for Flink users so that they can customize their deployment. In particular next settings seems to be the minimum required to adjust GCS timeout with Job's checkpoint config:

• maxAttempts
• initialRpcTimeout
• rpcTimeoutMultiplier
• maxRpcTimeout
• totalTimeout

 

Basically the proposal is to be able to tune the timeout via multiplier, maxAttemts + totalTimeout mechanisms.

All of the config options should be optional and the default one should be used in case some of configs are not provided.

Issue Links

• relates to: FLINK-32877 Support for HTTP connect and timeout options while writes in GCS connector
• links to: GitHub Pull Request #24753