[HDDS-7593] Supporting HSync and lease recovery

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 05/Dec/22 18:53
• Updated: 20/Nov/23 19:35

Description

This is the umbrella jira encompassing the design and implementation of hflush API and lease recovery support in Ozone. This feature enables new use cases such as HBase and Solr where the Write Ahead Log and Transaction Logs are flushed to Ozone constantly.

A design doc will be added shortly.

Issue Links

• causes: HDDS-8198 Hbase on Ozone has very high latency while running IntegrationTestIngestWithTags
• causes: HDDS-9146 Potential data loss with HSync due to deletedTable entry having the same block as keyTable entry's
• causes: HDDS-8199 "Directory is not empty" while running Hbase(configured to run on Ozone) LoadTestTool
• causes: HDDS-8200 [Hbase-Ozone] "RegionTooBusyException: Failed to obtain lock" while running IntegrationTestIngestWithTags
• is a parent of: HDDS-9164 [Hsync] moves blocks to deleted table on final commit
• is duplicated by: HDDS-4353 Ozone supports hflush operation
• is duplicated by: HDDS-633 HBase service fails to start with OzoneFS as defaultFS
• is required by: HBASE-27740 Support Ozone as a WAL backing storage
• relates to: HDDS-8047 [hsync] Make Putblock performance acceptable
• relates to: HDDS-7722 FSO buckets fail to invalidate open file table cache when committing a key
• relates to: HDDS-4353 Ozone supports hflush operation
• relates to: RATIS-1886 AppendLog sleep fixed time cause significant drop in write throughput
• relates to: HDDS-8825 Use apache/hadoop 3.3.5 docker image
• Testing discovered: HDDS-8024 When readChunk from a datanode fails, retry other datanodes.
• Testing discovered: HDDS-8198 Hbase on Ozone has very high latency while running IntegrationTestIngestWithTags
• Testing discovered: HDDS-8932 [Hbase-Ozone] Hbase on Ozone EC Should gracefully exit
• Testing discovered: HDDS-8969 [Phoenix-Ozone] Failed to identify the fs No FileSystem for scheme "ofs"
• Testing discovered: HDDS-8292 Inconsistent key name handling for FSO bucket files
• Testing discovered: HDDS-8267 getOMDBUpdates requests crashed Ozone Manager
• Testing discovered: HDDS-8537 [Hbase-Ozone] OMKeyRenameRequestWithFSO: Rename key failed
• Testing discovered: HDDS-8090 When getBlock from a datanode fails, retry other datanodes.

Sub-Tasks

• recoverLease should close underlying streams: Open
• [hsync] Reject new allocation block for file under lease recovery: Open
• [hsync] Add CLI to list all the open files: In Progress
• [hsync] Combine WriteData and PutBlock requests into one: In Progress
• [hsync] Make Putblock performance acceptable: Open
• [File Lease] Client side lease renewer thread and request message: In Progress
• [File Lease] OM adds request message handler: In Progress
• [File Lease] OM adds FileLeaseManager: In Progress
• [hsync] write after lease recovery does not fail: Open
• two client parallel perform commit with Hsync can cause dataloss: Open
• [hsync] DataNode to deserialize Ratis transaction only once: Open
• [hsync] Update OpenTelemetry traces in the write path: In Progress
• [hsync] Flush to only wait for majority of DataNodes: Open
• [hsync] Redesign the lease recovery protocol so block length is updated correctly at OM: Open
• [hsync] Skip updating block length at OM during hsync: Open
• [hsync] disk usage thread aborts if ratis log rolls very quickly: Open
• Add hsync and lease recovery Documentation: Open
• [hsync] Make Putblock performance acceptable - Skeleton code: Open
• [hsync] Make Putblock performance acceptable - DataNode side: Open
• [hsync] Make Putblock performance acceptable - Client side: Open
• [hsync] Make Putblock performance acceptable - Tool cleanup, guardrails: Open
• [hsync] Checking disk usage at every write request is expensive for HBase: Open
• [hsync] Unsynchronize hsync API: Open
• [hsync] Support renaming open files: Open