[HUDI-5835] spark cannot read mor table after execute update statement

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 0.13.0
• Fix Version/s: 0.13.1
• Component/s: spark
• Labels: pull-request-available
• Assignee: Tao Meng
• Reporter: Tao Meng
• Created: 23/Feb/23 10:24
• Updated: 22/May/23 09:13

Description

avro schema create by sparksql miss avro name and namespace, 

This will lead the read schema and write schema of the log file to be incompatible

 

// code placeholder
 spark.sql(
   s"""
      |create table $tableName (
      |  id int,
      |  name string,
      |  price double,
      |  ts long,
      |  ff decimal(38, 10)
      |) using hudi
      | location '${tablePath.toString}'
      | tblproperties (
      |  type = 'mor',
      |  primaryKey = 'id',
      |  preCombineField = 'ts'
      | )
""".stripMargin)
 spark.sql(s"insert into $tableName select 1, 'a1', 10, 1000, 10.0")
checkAnswer(s"select id, name, price, ts from $tableName")(
  Seq(1, "a1", 10.0, 1000)
)
spark.sql(s"update $tableName set price = 22 where id = 1")
checkAnswer(s"select id, name, price, ts from $tableName")(   -------- failed
  Seq(1, "a1", 22.0, 1000)
)

 

Issue Links

• links to: GitHub Pull Request #8026