[HUDI-4975] datahub sync bundle causes class loading issue

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Component/s: dependencies
• Labels: pull-request-available
• Story Points: 1
• Epic Link: Improve dependencies
• Assignee: Shiyan Xu
• Reporter: Shiyan Xu
• Created: 03/Oct/22 18:27
• Updated: 10/Oct/22 08:45
• Resolved: 10/Oct/22 08:45

Description

run utilities-slim.jar as the main jar for deltastreamer

set --jars /tmp/hudi-datahub-sync-bundle-0.12.1-rc1.jar,/tmp/hudi-spark3.1-bundle_2.12-0.12.1-rc1.jar

put datahub sync bundle before spark bundle resulted in class loader issue. works fine if spark bundle goes first

Caused by: java.lang.NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation
	at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:78)
	at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:70)
	at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:54)
	at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:104)
	at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:76)
	at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:46)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
	at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
	at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:135)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 14 more

Issue Links

• is related to: HUDI-4971 aws bundle causes class loading issue
• links to: GitHub Pull Request #6896