[HUDI-898] Need to add Schema parameter to HoodieRecordPayload::preCombine

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Unresolved
• Component/s: Common Core
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Yixue Zhu
• Created: 14/May/20 19:26
• Updated: 07/Dec/21 23:34

Description

We are working on Mongo Oplog integration with Hudi, to stream Mongo updates to Hudi tables.

There are 4 Mongo OpLog operations we need to handle, CRUD (create, read, update, delete).

Currently Hudi handle create/read, delete, but not update well with existing preCombine API in HoodieRecordPayload class. In particularly, Update operation contains "patch" field, which is extended Json describing update for dot separated field paths.

We need to pass Avro schema to preCombine API for it to work:

Even though BaseAvroPayload constructor accepts GenericRecord, which has Avro schema reference, but it materialize GenericRecord to bytes, to support serialization/deserialization by ExternalSpillableMap.

 

Is there concern/objection to this? in other words, have I overlooked something?

 

Issue Links

• links to: GitHub Pull Request #2141