[HUDI-1451] Support bulk insert v2 with Spark 3

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Labels: pull-request-available
• Assignee: Wenning Ding
• Reporter: Wenning Ding
• Created: 11/Dec/20 07:24
• Updated: 26/Dec/20 01:02
• Resolved: 26/Dec/20 01:02

Description

Recently, we made Hudi support Spark 3.0.0: https://github.com/apache/hudi/pull/2208. However, there was a large refactor done to Spark datasource V2 API interfaces from version 2.4.4 → 3.0.0. So currently our bulk insert v2(https://issues.apache.org/jira/browse/HUDI-1013) feature is marked as unsupported with Spark3.

We need to redesign the bulk insert v2 part with datasource v2 api in Spark 3.0.0.

Issue Links

• links to: GitHub Pull Request #2328