[HUDI-3936] When precombine field is nested, MOR query breaks

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: spark
• Labels: pull-request-available
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: sivabalan narayanan

Reviewers

Alexey Kudinkin, Shiyan Xu

• Created: 21/Apr/22 01:52
• Updated: 22/Apr/22 02:04
• Resolved: 22/Apr/22 02:04

Description

For MOR table, when using a nested field as precombined key, when querying the table after upserts, the following exception is thrown:

 

Exception in thread "main" java.util.NoSuchElementException: key not found: meta.version
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at org.apache.hudi.HoodieSparkUtils$.$anonfun$getRequiredSchema$2(HoodieSparkUtils.scala:325)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
	at org.apache.hudi.HoodieSparkUtils$.getRequiredSchema(HoodieSparkUtils.scala:325)
	at org.apache.hudi.HoodieBaseRelation.buildScan(HoodieBaseRelation.scala:219)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.$anonfun$apply$4(DataSourceStrategy.scala:332)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.$anonfun$pruneFilterProject$1(DataSourceStrategy.scala:365)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.pruneFilterProjectRaw(DataSourceStrategy.scala:442)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.pruneFilterProject(DataSourceStrategy.scala:364)
	at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.apply(DataSourceStrategy.scala:332)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491) 

Issue Links

• is duplicated by: HUDI-3937 Fix NoSuchElementException with nest field as preCombine key in MOR
• links to: GitHub Pull Request #5379