[HUDI-6627] Spark write client fails when write schema is null

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Labels: pull-request-available
• Story Points: 1
• Assignee: Unassigned
• Reporter: Vinish Reddy
• Created: 01/Aug/23 14:19
• Updated: 03/Aug/23 01:09
• Resolved: 03/Aug/23 01:09

Description

When source returns an empty option in deltastreamer, the writer schema is null. This causes an NPE with the table schema validation in spark write client causing the below exception. We should skip this validation when writer schema is null. 

org.apache.hudi.exception.HoodieInsertException: Failed insert schema compability check.
	at org.apache.hudi.table.HoodieTable.validateInsertSchema(HoodieTable.java:851)
	at org.apache.hudi.client.SparkRDDWriteClient.insert(SparkRDDWriteClient.java:185)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:690)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:396)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.ingestOnce(HoodieDeltaStreamer.java:876)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at com.onehouse.hudi.OnehouseDeltaStreamer$MultiTableSyncService.lambda$null$1(OnehouseDeltaStreamer.java:319)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: Failed to read schema/check compatibility for base path s3a://onehouse-customer-bucket-2451e78f/data-lake/chandra_data_lake_default/xml_flatten_struct_test
	at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:830)
	at org.apache.hudi.table.HoodieTable.validateInsertSchema(HoodieTable.java:849)
	... 10 more
Caused by: java.lang.NullPointerException
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1158)
	at org.apache.avro.Schema$Parser.parse(Schema.java:1418)
	at org.apache.hudi.avro.HoodieAvroUtils.createHoodieWriteSchema(HoodieAvroUtils.java:302)
	at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:826)
	... 11 more

 

Issue Links

• links to: GitHub Pull Request #9335