[HUDI-4735] Spark2 bundles made from master after 2022-07-23 failed to stop

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: hive, spark, spark-sql
• Assignee: Shiyan Xu
• Reporter: Vitali Obukh
• Created: 29/Aug/22 15:32
• Updated: 21/Dec/23 15:28

Description

Daily Spark2 Structured streaming application built against the latest change on master started failing on 2022-07-23 with the error:

22/07/28 10:52:50 ERROR org.apache.hudi.HoodieStreamingSink: Micro batch id=2 threw following exception:
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'test' not found;

Actually the database exists (created by the Hudi). Spark3 application with the same code works fine.
Hudi options used for writing:

 Map("hoodie.datasource.write.operation" -> "upsert",
  "hoodie.datasource.write.table.type" -> "MERGE_ON_READ",
  "hoodie.datasource.write.recordkey.field" -> s"$Meta.$Key",
  "hoodie.datasource.write.precombine.field" -> s"$Meta.$Version",
  "hoodie.datasource.write.payload.class" -> "com.walmart.udps.hudi.integration.HudiTestAvroPayload",
  "hoodie.compaction.payload.class" -> "com.walmart.udps.hudi.integration.HudiTestAvroPayload",
  "hoodie.datasource.write.partitionpath.field" -> TopLevelPartition,
  "hoodie.datasource.write.hive_style_partitioning" -> "true",
  "hoodie.datasource.hive_sync.enable" -> "true",
  "hoodie.datasource.hive_sync.mode" -> "hms",
  "hoodie.datasource.hive_sync.database" -> dbName(),
  "hoodie.datasource.hive_sync.partition_fields" -> TopLevelPartition,
  "hoodie.datasource.compaction.async.enable" -> "false"
)

Cloud platform - Google Dataproc.
Spark version - 3.1.3.