[HDDS-5908] MPU getKey can fail, if completeMPU result is still in cache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Labels: pull-request-available
• Assignee: Bharat Viswanadham
• Reporter: Ethan Rose
• Created: 28/Oct/21 17:28
• Updated: 04/Nov/21 19:43
• Resolved: 02/Nov/21 17:24

Description

Failure was observed on this CI run: https://github.com/apache/ozone/runs/4015387580?check_suite_focus=true

The output bundles expired before I could add them to this Jira, but the failure can be reproduced on master by applying the patch attached to this Jira and running the test. The patch speeds up repeated execution of the test by repeatedly writing keys without having to spin up a new mini ozone cluster in between each write.  It usually takes about 4 minutes and 1200 iterations to reproduce.

The failing assertion is:

OzoneInputStream inputStream = bucket.readKey(keyName);
Assert.assertTrue(inputStream instanceof MultipartCryptoKeyInputStream);

Indicating a plain OzoneInputStream is returned, since the class has no other sub classes.

I have found the reason for this.
1. If complete MPU is completed, it adds the entry to keyTable.
2. Now if getKey happens on this, if doublebuffer flush not completed flush and cleaned up cache if entry is still in keyTable, the key info returned as Not Mpu Key, this is due to a bug in OmKeyInfo#copyObject Which is not using isMultipartKey.

Issue Links

• is related to: HDDS-4653 Support TDE for MPU Keys on Encrypted Buckets
• links to: GitHub Pull Request #2787