[HUDI-1667] Fix bug when HoodieMergeOnReadRDD read record from base file, Hoodie may set non-null value in field which is null if vectorization is enabled.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Common Core
• Labels: pull-request-available
• Assignee: Lietong Liu
• Reporter: Lietong Liu
• Created: 05/Mar/21 09:26
• Updated: 20/Aug/24 01:15
• Resolved: 21/Mar/21 10:23

Description

When HoodieMergeOnReadRDD read record from base file,  will create new InternalRow base on requiredStructSchema.

//代码占位符
private def createRowWithRequiredSchema(row: InternalRow): InternalRow = {
  val rowToReturn = new SpecificInternalRow(tableState.requiredStructSchema)
  val posIterator = requiredFieldPosition.iterator
  var curIndex = 0
  tableState.requiredStructSchema.foreach(
    f => {
      val curPos = posIterator.next()
      val curField = row.get(curPos, f.dataType)
      rowToReturn.update(curIndex, curField)
      curIndex = curIndex + 1
    }
  )
  rowToReturn
}

 Hoodie doesn't check isNull when get value from all fields here.

If vectorization is enabled, which  means row is ColumnarBatchRow.  **ColumnarBatchRow may return non-null value even if value of field is null. So, hoodie may set non-null value in field which is null.

Issue Links

• links to: GitHub Pull Request #2636
• links to: GitHub Pull Request #2636