[HUDI-8489] Fix encoding of secondary index key

• Type: Task
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Epic Link: secondary index
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit
• Created: 07/Nov/24 15:36
• Updated: 5 days ago 03:09

Description

Secondary index key is a combination of secondaryKey and recordKey. There are two ways to encode with a delimiter ($):

1. Run base64 encoding: `Base64.encode(secondaryKey) + DELIMITER + Base64.encode(recordKey)`.  Base64 does not map to $. So, this gives us a neat and standard way to encode. Might not be very efficient for long strings? But, base64 is a standard scheme.
2. Escape special characters:  `escapeSpecialChars(secondaryKey) + DELIMITER + escapeSpecialChars(recordKey)`. The keys are readable and preserves the order. This is a custom scheme not used in other systems.

Ran a benchmark to compare encoding/decoding time and did not find much difference - https://gist.github.com/codope/b1c73abed748d77c0b4db974d835f9da