[HUDI-538] [UMBRELLA] Restructuring hudi client module for multi engine support

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Code Cleanup
• Labels: hudi-umbrellas
• Assignee: vinoyang
• Reporter: vinoyang
• Created: 16/Jan/20 06:57
• Updated: 09/Feb/21 19:08

Description

Hudi is currently tightly coupled with the Spark framework. It caused the integration with other computing engine more difficult. We plan to decouple it with Spark. This umbrella issue used to track this work.

Issue Links

• blocks: HUDI-184 Integrate Hudi with Apache Flink
• is blocked by: HUDI-309 General Redesign of Archived Timeline for efficient scan and management

Sub-Tasks

• Introduce a new pom module named hudi-writer-common: Resolved
• Restructure code/packages to move more code back into hudi-writer-common: Closed
• Decouple HoodieReadClient and AbstractHoodieClient to break the inheritance chain: Resolved
• Make ClientUtils spark-free: New
• Make HoodieCommitArchiveLog spark free: New
• Make HoodieWriteConfig spark free: Closed
• Make EmbeddedTimelineService spark free: New
• Remove unnecessary use of spark in savepoint timeline: Closed
• Make config package spark free: Closed
• Make io package spark free: Closed
• Remove the dependency of EmbeddedTimelineService from HoodieReadClient: Closed
• Move HoodieReadClient into hudi-spark module: New
• Prototype classes/abstractions to encapsule SparkContext and RDD: New
• Explore support for plugging in Spark data source V2: New
• Replace JavaSparkContext/SQLContext with SparkSession: Open
• Make AbstractHoodieClient spark-free: New
• Make CompactionAdminClient spark-free: New
• Ability to do small file handling without need for caching: In Progress
• Some classes that rely on JavaSparkContext to obtain Configuration can make them Spark-free.: Resolved
• Remove spark context in ClientUtils and HoodieIndex: Closed
• Replace part of spark context by hadoop configuration in HoodieTable: Closed
• Replace part of Spark context by hadoop configuration in AbstractHoodieClient: Closed
• Replace jsc.hadoopConfiguration by hadoop configuration in hudi-client testcase: Closed