[HUDI-4971] aws bundle causes class loading issue

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Component/s: dependencies
• Labels: pull-request-available
• Story Points: 1
• Epic Link: Improve dependencies
• Assignee: Shiyan Xu
• Reporter: Shiyan Xu
• Created: 03/Oct/22 12:09
• Updated: 23/Oct/22 06:45
• Resolved: 10/Oct/22 08:03

Description

Putting aws bundle first before spark bundle will cause class loading issue

Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.objenesis.strategy.InstantiatorStrategy
  at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

Reprod with quickstart example.

spark-shell \
--jars /tmp/spark31/hudi-aws-bundle-0.12.1-rc1.jar,/tmp/spark31/hudi-spark3.1-bundle_2.12-0.12.1-rc1.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'

// spark-shell
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.common.model.HoodieRecord

val tableName = "hudi_trips_cow"
val basePath = "file:///tmp/hudi_trips_cow"
val dataGen = new DataGenerator
val snapshotQuery = "SELECT begin_lat, begin_lon, driver, end_lat, end_lon, fare, partitionpath, rider, ts, uuid FROM hudi_ro_table"

val inserts = convertToStringList(dataGen.generateInserts(10))
val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
df.write.format("hudi").
  options(getQuickstartWriteConfigs).
  option(PRECOMBINE_FIELD_OPT_KEY, "ts").
  option(RECORDKEY_FIELD_OPT_KEY, "uuid").
  option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
  option(TABLE_NAME, tableName).
  mode(Overwrite).
  save(basePath)

Issue Links

• relates to: HUDI-4975 datahub sync bundle causes class loading issue
• links to: GitHub Pull Request #6873
• links to: GitHub Pull Request #7014