[HUDI-4850] Implement DeltaStreamer Source for Google Cloud Storage

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.13.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Epic Link: Implement DeltaStreamer Source for cloud object stores
• Assignee: Unassigned
• Reporter: Pramod Biligiri
• Created: 15/Sep/22 17:11
• Updated: 30/Sep/22 07:07
• Resolved: 30/Sep/22 07:06

Description

It should be possible to reliably ingest data from GCS buckets into Hudi using a Deltastreamer Source. Such a feature already exists to ingest from AWS S3 buckets, as discussed in HUDI-1897 and described in a Hudi blog post: https://hudi.apache.org/blog/2021/08/23/s3-events-source/

Issue Links

• duplicates: HUDI-1898 Implement DeltaStreamer Source for GCS
• is a parent of: HUDI-4927 GCS Ingestion supports only new file uploads, no deletion and repeated uploads
• is a parent of: HUDI-4931 Explore fat jar option for gcs-connector lib used during GCS Ingestion
• links to: GitHub Pull Request #6665