[HUDI-242] [RFC-12] Support Efficient bootstrap of large parquet datasets to Hudi

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Usability
• Labels: pull-request-available
• Assignee: Vinoth Chandar
• Reporter: Balaji Varadarajan
• Created: 10/Sep/19 18:54
• Updated: 25/Jan/21 20:41
• Resolved: 01/Sep/20 19:47

Description

 Support Efficient bootstrap of large parquet tables

Issue Links

• is a parent of: HUDI-1102 Separate out Spark and Path detection utilities used in Bootstrap datasource work
• is a parent of: HUDI-1051 Improve MOR datasource reader file listing and path handling
• links to: GitHub Pull Request #1678
• links to: GitHub Pull Request #1876

Sub-Tasks

• Basic Implementation for verifying if bootstrapping works end to end: Closed
• Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations: Closed
• Bootstrap Index - Implementation: Closed
• Cleanup bootstrap code and create PR for FileStystemView changes: Closed
• Cleanup bootstrap code and create write APIs for supporting bootstrap: Closed
• Implement Hive Query Side Integration for querying tables containing bootstrap file slices: Closed
• Implement support for bootstrapping in HoodieDeltaStreamer: Closed
• Implement Spark DataSource Support for querying bootstrapped tables: Resolved
• Automated end to end Integration Test: Closed
• Implement upsert functionality for handling updates to these bootstrap file slices: Closed
• Implement CLI support for performing bootstrap: Resolved
• Long Running Testing to certify Bootstrapping: Resolved
• Hive Sync Integration of bootstrapped table: Closed
• Implement support for bootstrapping via Spark datasource API: Closed
• Spark DS Support for incremental queries for bootstrapped tables: Closed
• Add a knob to change partition-path style while performing metadata bootstrap: Closed
• Metadata Bootstrap Key Generator needs to handle complex keys correctly: Closed
• Metadata Bootstrap Query Testing Master TIcket: Closed
• Test COW : Hive Read Optimized Query with metadata bootstrap: Resolved
• Test MOR : Hive Read Optimized Query with metadata bootstrap: Resolved
• Test MOR : Spark SQL Read Optimized Query with metadata bootstrap: Resolved
• Test MOR : Spark SQL Realtime Query with metadata bootstrap: Resolved
• Test MOR : Hive Realtime Query with metadata bootstrap: Resolved
• Test COW : Spark SQL Read Optimized Query with metadata bootstrap: Resolved
• Test COW : Spark Data Source Read Optimized Queries: Resolved
• Web documentation for explaining how to bootstrap: Closed
• Open Questions before merging Bootstrap: Resolved
• Support for cleaning source data: Resolved
• Fix HFileBootstrapIndexReader.getIndexedPartitions() returns unclean partition name: Closed
• Bootstrap Implementation Bugs: Resolved
• Parallelize listing of Source dataset partitions: Resolved
• Address performance issues with finalizing writes on S3: Closed
• Separate out Spark and Path detection utilities used in Bootstrap datasource work: Closed
• Hudi changes for bootstrapped tables integration with Presto: Resolved