[HUDI-7609] Spark cannot write the hudi table containing array type created by flink

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.15.0, 1.0.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: 陈磊
• Created: 12/Apr/24 11:09
• Updated: 16/Apr/24 00:42

Description

When flink creates a Hudi table containing an array field, the elements of the default array field cannot be nullable. When using Spark SQL to read data from the Hive table to the Hudi table, a field verification exception will occur.

2024-03-27 12:47:51 INFO org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'decentral_level1
2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:138)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:549)
2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
2024-03-27 12:47:51 INFO at scala.collection.immutable.List.foreach(List.scala:392)
2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.map(TraversableLike.scala:238)
2024-03-27 12:47:51 INFO at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
2024-03-27 12:47:51 INFO at scala.collection.immutable.List.map(List.scala:298)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.types.StructType$.fromAttributes(StructType.scala:549)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.plans.QueryPlan.schema$lzycompute(QueryPlan.scala:281)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.catalyst.plans.QueryPlan.schema(QueryPlan.scala:281)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.alignQueryOutput(InsertIntoHoodieTableCommand.scala:153)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:105)
2024-03-27 12:47:51 INFO at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:60) 

Issue Links

• links to: GitHub Pull Request #11006