[HUDI-254] Provide mechanism for installing hudi-spark-bundle onto an existing spark installation

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: spark
• Assignee: Vinoth Chandar
• Reporter: Vinoth Chandar
• Created: 17/Sep/19 00:15
• Updated: 26/Dec/19 00:43
• Resolved: 17/Sep/19 21:05

Description

A lot of discussions around this kicked off from https://github.com/apache/incubator-hudi/issues/869 

Breaking down into phases, when we drop the hudi-spark-bundle*.jar onto the `jars` folder 

 

a) Writing data via Hudi datasource should work 

b) Spark datasource reads should work

 

c)  a + Hive Sync should work

d) SparkSQL on Hive synced table works 

 

Start with Spark 2.3 (current demo setup) and then proceed to 2.4 and iron out issues.

 

Issue Links

• relates to: HUDI-260 Hudi Spark Bundle does not work when passed in extraClassPath option
• links to: GitHub Pull Request #903