[HUDI-1443] Remove record deserialization in RDDCustomColumnsSortPartitioner

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: performance
• Epic Link: Table Services - Clustering
• Assignee: Unassigned
• Reporter: satish
• Created: 09/Dec/20 01:11
• Updated: 06/Jan/22 23:25
• Resolved: 22/Dec/20 01:37

Description

https://github.com/apache/hudi/pull/2263#discussion_r533653930 has the context. When sorting is specified as part of clustering, we use custom partitioner RDDCustomColumnsSortPartitioner. This deserializes schema to get values for sort columns. Check if its possible to avoid this and implement the suggestion in PR.

We tried another approach by adding SerializableSchema. But this is not working for nested schemas. See test failing here. Fix this serialization and use it in RDDCustomColumnsSortPartitioner