[HDDS-8095] Unbuffer not supported in TDE buckets

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Target Version/s: 1.3.1
• Assignee: Wei-Chiu Chuang
• Reporter: Michael Smith
• Created: 07/Mar/23 17:05
• Updated: 26/Jun/23 23:43
• Resolved: 09/Mar/23 16:18

Description

Reading data (with Impala using file handle caching) from a bucket with transparent data encryption (TDE) enabled fails to unbuffer the input stream after read is complete.

Impala attempts hdfsUnbufferFile on cached file handles after reading is done. Previous tickets (HDDS-4320, HDDS-7135) have addressed issues with this, and been tested to work correctly. However when we enable TDE, unbuffering stops working again.

Enabling debug logging on org.apache.hadoop shows

I0307 09:00:58.513293 31969 StreamCapabilitiesPolicy.java:50] 4c4de8d99785e5c8:a97f7edb00000000] org.apache.hadoop.ozone.client.io.KeyInputStream: does not implement StreamCapabilities and the unbuffer capability

Issue Links

• is related to: HDDS-8113 Remove Hadoop 2.7 compatibility hack once Hadoop 2.7 support is dropped
• relates to: HDDS-7135 ofs file input stream should support StreamCapabilities interface
• relates to: HDDS-4320 Let Ozone input streams implement CanUnbuffer
• links to: GitHub Pull Request #4361