[HUDI-3995] Bulk insert row writer perf improvements

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.1
• Component/s: spark, writer-core
• Labels: pull-request-available
• Story Points: 3
• Epic Link: Performance Improvements
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 28/Apr/22 18:59
• Updated: 12/Jul/22 14:43
• Resolved: 11/May/22 01:06

Description

EDIT
====

While investigating, perf hits in the Bulk Insert a few issues were found:

1. NonPartitionedKeyGenerator does not implement `getRecordKey`, `getParititionKey` for `InternalRow`, leading to invocation of default implementation converting row to Avro.
2. HUDI-3993: Using UDF to fetch record keys, similarly has to deserialize `InternalRow` into deserialized `Row`

 

Issue Links

• is related to: HUDI-3993 Avoid calling into Spark UDF in Bulk Insert
• is related to: HUDI-4038 Avoid invoking `getDataSize` in the hot-path
• is related to: HUDI-4039 Make sure builtin key-generators can efficiently fetch record-key, partition-path
• relates to: HUDI-4384 Hive style partition not work and record key loss prefix using ComplexKey in bulk_insert
• links to: GitHub Pull Request #5462