[HUDI-2438] [Umbrella] [RFC-34] Implement BigQuerySyncTool for BigQuery Sync

• Type: Epic
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.11.0
• Component/s: Common Core, meta-sync
• Labels: BigQuery Integration pull-request-available
• Story Points: 0
• Epic Name: Hudi-BigQuery
• Assignee: Vinoth Govindarajan
• Reporter: Vinoth Govindarajan

Reviewers

Shiyan Xu, Vinoth Chandar

• Created: 15/Sep/21 21:30
• Updated: 28/Apr/22 13:23
• Resolved: 28/Apr/22 13:23

Description

BigQuery is Google Cloud's fully managed, petabyte-scale, and cost-effective analytics data warehouse that lets you run analytics over vast amounts of data in near real-time. BigQuery currently doesn’t support Apache Hudi file format, but it has support for the Parquet file format. The proposal is to implement a BigQuerySync similar to HiveSync to sync the Hudi table as the BigQuery External Parquet table so that users can query the Hudi tables using BigQuery. Uber is already syncing some of its Hudi tables to BigQuery data mart this will help them to write, sync, and query.

 

More details are in RFC-34: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188745980

Issue Links

• links to: GitHub Pull Request #5218