[BEAM-13734] Support cache directories that use GCS buckets

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Implemented
• Fix Version/s: 2.37.0
• Component/s: runner-py-interactive
• Labels: stale-assigned
• Assignee: Victor Chen
• Reporter: Victor Chen
• Created: 24/Jan/22 23:36
• Updated: 01/Apr/22 19:43
• Resolved: 04/Mar/22 19:35

Description

• Builds off of the work accomplished under BEAM-13685
• Modified interactive_environment.py to support caching to a bucket on GCS for batch processing pipelines
• If a specified bucket does not exist, the pipeline will terminate and return an error specifying that the bucket does not exist
• Added cleanup() functionality to cache_manager.py, to enable the FileBasedCacheManager class to automatically delete cached values on GCS when a bucket path is specified
• Added docstring to interactive_beam.py with an example of GCS path assignment
• Cached files on GCS will be stored under a directory represented by the value of id(pipeline).
  • Example cached path: gs://my-gcs-bucket/cache/dir/id(pipeline)

Issue Links

• links to: GitHub Pull Request #16601