[HUDI-8584] Unify the unsupported types across SPARK and AVRO record type

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.1
• Component/s: metadata
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan

Reviewers

Sagar Sumit

• Created: 26/Nov/24 23:37
• Updated: 20 hours ago
• Resolved: 20 hours ago

Description

As of now, here is what our unsupported types look like 

private static boolean canCompare(Schema schema, HoodieRecordType recordType) {
  // if recordType is SPARK then we cannot compare RECORD and ARRAY types in addition to MAP type
  if (recordType == HoodieRecordType.SPARK) {
    return schema.getType() != Schema.Type.RECORD && schema.getType() != Schema.Type.ARRAY && schema.getType() != Schema.Type.MAP;
  }
  return schema.getType() != Schema.Type.MAP;
} 

Lets unify the exp here. 

we can make MAP, ARRAYS, RECORD as unsupported types across both record types. Eventually we might move to SPARK record type. So, anyways those are not supported at the moment. 

 

 

 

Issue Links

• links to: GitHub Pull Request #12364