[HUDI-841] Abstract common meta sync module support multiple meta service

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: hive
• Assignee: liwei
• Reporter: liwei
• Created: 27/Apr/20 11:26
• Updated: 10/Dec/20 15:27
• Resolved: 06/Aug/20 06:24

Description

Currently Hudi only supports sync dataset metadata to Hive through hive jdbc and IMetaStoreClient. When you need to sync to other frameworks, such as aws glue, aliyun DataLake analytics, etc. You need to copy a lot of code from HoodieHiveClient, which creates a lot of redundant code. So need to redesign the hudi-hive-sync module to support other frameworks and reuse current code as much as possible. Only the interface is provided by Hudi, and the implement is customized by different services as hive 、aws glue、aliyun DataLake analytics.

Issue Links

• is a parent of: HUDI-1342 hudi-dla-sync support modify table properties
• links to: RFC - 17 Abstract common meta sync module support multiple meta service

Sub-Tasks

• Introduce a new pom module named hudi-common-sync: Resolved
• Restructure code/packages to implement hudi-common-sync: Resolved
• Restructure hudi-hive-sync implement hudi-common-sync: Resolved
• hudi-spark support the multi catalog conf: Resolved
• hudi-utilities add the hudi-common-sync metrics: Resolved
• support aliyun DLA meta: Closed
• hudi dla sync skip rt create: Closed