[BEAM-12818] When writing to GCS, spread prefix of temporary files and reuse autoscaling of the temporary directory

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.35.0
• Component/s: io-java-gcp
• Assignee: Sam Whittle
• Reporter: Sam Whittle
• Created: 30/Aug/21 09:53
• Updated: 17/Nov/21 19:43
• Resolved: 17/Nov/21 19:33

Description

When writing files using FileIO, the given temporary directory has a subdirectory created in it for each FileBasedSink. This is useful for non-windowed output where the temporary directory can be matched to delete leftover files that were lost during processing.

However for windowed writes such subdirectories are unnecessary and cause a common prefix to be shared for the temporary files. Additionally this common prefix varies per job and thus the autoscaling for the previous prefix is no longer effective, see
https://cloud.google.com/storage/docs/request-rate#randomness_after_sequential_prefixes_is_not_as_effective

Issue Links

• links to: GitHub Pull Request #15416