[HUDI-667] HoodieTestDataGenerator does not delete keys correctly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: Prashant Wason
• Created: 06/Mar/20 19:48
• Updated: 13/Mar/20 16:32
• Resolved: 13/Mar/20 16:32

Description

HoodieTestDataGenerator is used to generate sample data for unit-tests. It allows generating HoodieRecords for insert/update/delete. It maintains the record keys in a HashMap.

private final Map<Integer, KeyPartition> existingKeys;

There are two issues in the implementation:

1. Delete from existingKeys uses KeyPartition rather than Integer keys
2. Inserting records after deletes is not correctly handled

The implementation uses the Integer key so that values can be looked up randomly. Assume three values were inserted, then the HashMap will hold:

0 -> KeyPartition1
1 -> KeyPartition2
2 -> KeyPartition3

Now if we delete KeyPartition2  (generate a random record for deletion), the HashMap will be:

0 -> KeyPartition1
2 -> KeyPartition3

 

Now if we issue a insertBatch() then the insert is existingKeys.put(existingKeys.size(), KeyPartition3) which will overwrite the KeyPartition3 already in the map rather than actually inserting a new entry in the map.

Issue Links

• links to: GitHub Pull Request #1395