[SPARK-38542] UnsafeHashedRelation should serialize numKeys out

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: SQL
• Labels: correctness
• Assignee: Unassigned
• Reporter: mcdull_zhang
• Created: 13/Mar/22 12:18
• Updated: 02/Sep/22 21:41
• Resolved: 16/Mar/22 06:18

Description

At present, UnsafeHashedRelation does not write out numKeys during serialization, so the numKeys of UnsafeHashedRelation obtained by deserialization is equal to 0. The numFields of UnsafeRows returned by UnsafeHashedRelation.keys() are all 0, which can lead to missing or incorrect data.

 

For example, in SubqueryBroadcastExec, the HashedRelation.keys() function is called.

val broadcastRelation = child.executeBroadcast[HashedRelation]().value
val (iter, expr) = if (broadcastRelation.isInstanceOf[LongHashedRelation]) {
  (broadcastRelation.keys(), HashJoin.extractKeyExprAt(buildKeys, index))
} else {
  (broadcastRelation.keys(),
    BoundReference(index, buildKeys(index).dataType, buildKeys(index).nullable))
}

 

Issue Links

• links to: [Github] Pull Request #35836 (mcdull-zhang)