[HDDS-7593] Supporting HSync and lease recovery

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 05/Dec/22 18:53
• Updated: 13/Jun/24 17:39

Description

This is the umbrella jira encompassing the design and implementation of hflush API and lease recovery support in Ozone. This feature enables new use cases such as HBase and Solr where the Write Ahead Log and Transaction Logs are flushed to Ozone constantly.

A design doc will be added shortly.

Issue Links

• causes: HDDS-9146 Potential data loss with HSync due to deletedTable entry having the same block as keyTable entry's
• causes: HDDS-8198 Hbase on Ozone has very high latency while running IntegrationTestIngestWithTags
• causes: HDDS-8199 "Directory is not empty" while running Hbase(configured to run on Ozone) LoadTestTool
• causes: HDDS-8200 [Hbase-Ozone] "RegionTooBusyException: Failed to obtain lock" while running IntegrationTestIngestWithTags
• Discovered while testing: HDDS-10536 [Hbase Ozone] BLOCK_TOKEN_VERIFICATION failed while long running YCSB
• is a parent of: HDDS-9164 [Hsync] moves blocks to deleted table on final commit
• is duplicated by: HDDS-4353 Ozone supports hflush operation
• is duplicated by: HDDS-633 HBase service fails to start with OzoneFS as defaultFS
• is related to: HDDS-11012 [Hbase-Ozone] HMaster down with NO_REPLICA_FOUND causing "CorruptHFileException: Problem reading HFile Trailer"
• is required by: HBASE-27740 Support Ozone as a WAL backing storage
• relates to: HDDS-7722 FSO buckets fail to invalidate open file table cache when committing a key
• relates to: HDDS-8047 [hsync] Make Putblock performance acceptable
• relates to: HDDS-4353 Ozone supports hflush operation
• relates to: HDDS-10465 Change ozone.client.bytes.per.checksum default to 16KB
• relates to: RATIS-1886 AppendLog sleep fixed time cause significant drop in write throughput
• relates to: RATIS-2089 Add CommitInfoProto in NotReplicatedException
• relates to: HDDS-8825 Use apache/hadoop 3.3.5 docker image
• supercedes: HDDS-3589 Support running HBase on Ozone.
• Testing discovered: HDDS-8024 When readChunk from a datanode fails, retry other datanodes.
• Testing discovered: HDDS-10860 Intermittent failure in TestLeaseRecovery.testFinalizeBlockFailure
• Testing discovered: HDDS-8969 [Phoenix-Ozone] Failed to identify the fs No FileSystem for scheme "ofs"
• Testing discovered: HDDS-10298 ozone admin om lof, total open files estimation is ambigious
• Testing discovered: HDDS-8292 Inconsistent key name handling for FSO bucket files
• Testing discovered: HDDS-8267 getOMDBUpdates requests crashed Ozone Manager
• Testing discovered: HDDS-8198 Hbase on Ozone has very high latency while running IntegrationTestIngestWithTags
• Testing discovered: HDDS-8537 [Hbase-Ozone] OMKeyRenameRequestWithFSO: Rename key failed
• Testing discovered: HDDS-8932 [Hbase-Ozone] Hbase on Ozone EC Should gracefully exit
• Testing discovered: HDDS-10190 [hsync] Lease recovery fails when file has just created without blocks
• Testing discovered: HDDS-10242 [hsync] Lease recovery failed as "Unable to find the block"
• Testing discovered: HDDS-10256 Block allocation should retry if SCM is in safe mode
• Testing discovered: HDDS-8090 When getBlock from a datanode fails, retry other datanodes.

Sub-Tasks

• Ozone client change to support HSync: Resolved
• Add support for application to probe output stream capability: Resolved
• Ozone Manager change to support HSync: Resolved
• hsync: Change KeyOutputStream to update length in OM: Resolved
• Potential discrepancy of key creation time may cause premature open key clean up: Resolved
• OM lease recovery for hsync'ed files: Resolved
• [hsync] Recon throws ClassCastException: Resolved
• [hsync] Outputstream in encrypted buckets do not return the correct stream capabilities: Resolved
• input stream does not refresh expired block token: Resolved
• Change Get Key Info to return HSync info: Resolved
• Ozone file systems to support Hadoop's PathCapabilities interface: Resolved
• Add hsync metrics in OM: Resolved
• Implement client initiated lease recovery: Resolved
• Add a flag to disable hsync by default: Resolved
• [hsync] KeyOutputStream is not thread safe: Resolved
• LeaseRecovery failing with NullPointer exception: Resolved
• [hsync] Add a CLI to recover lease: Resolved
• OM crash with NPE in OMKeyCommitRequest due to missing user info: Resolved
• Quota needs to be updated correctly for Hsync: Resolved
• TestHSync is no longer flaky: Resolved
• [hsync] reject renaming open file: Resolved
• O3fs/ofs to support setTimes() API: Resolved
• [hsync] OMKeyRequest: Detect allocated but uncommitted blocks: Resolved
• Disallow overwriting a hsync'ed key: Resolved
• Support setSafeMode(), isFileClosed() FileSystem API: Resolved
• [hsync] HBase RegionServer input stream not shut down properly: Resolved
• OM to reject hsync if ozone.fs.hsync.enabled is false: Resolved
• [hsync] A freon tool to focus on hsync/hflush performance: Resolved
• ozone freon --server is broken by HDDS-6176: Resolved
• ChunkInputStream should use new token after pipeline refresh: Resolved
• [hsync] File recovery support in OM: Resolved
• [hsync] File recovery support in Client: Resolved
• hsync: Interface to retrieve block info and finalize block in DN through ratis: Resolved
• [hsync] DataNode to deserialize Ratis transaction only once: Resolved
• Make recoverLease call idempotent: Resolved
• [hsync] Make Putblock performance acceptable - Skeleton code: Resolved
• [hsync] Cache serialized block token in output stream to reduce heap consumption: Resolved
• Introduce soft limit support for lease recovery: Resolved
• Add admin CLI to list open files: Resolved
• [hsync] Redesign the lease recovery protocol so block length is updated correctly at OM: Resolved
• [hsync] Rebase HDDS-7593 branch onto master: Resolved
• [hsync] write after lease recovery does not fail: Resolved
• [hsync] Make Putblock performance acceptable - DataNode side: Resolved
• [hsync]Support hard limit and auto recovery for hsync file: Resolved
• [hsync] Reduce updating block length times at OM during hsync: Resolved
• two client parallel perform commit with Hsync can cause dataloss: Resolved
• Deleted file reappears after HSync: Resolved
• Add hsync metadata to hsync'ed keys in OpenKeyTable as well: Resolved
• Migrate tests to JUnit5: Resolved
• [hsync] Make Putblock performance acceptable - Client side: Resolved
• Merge recent commits from master to HDDS-7593: Resolved
• [hsync] Make Putblock performance acceptable: Resolved
• DataNode doesn't set proper DatanodeVersion when registering with SCM: Resolved
• [hsync] Output stream should support direct byte buffer: Resolved
• [hsync] disk usage thread aborts if ratis log rolls very quickly: Resolved
• [hsync] Revisit configuration keys for incremental chunk list after HDDS-9884: Resolved
• [hsync] MockDatanodeStorage.writeChunk should make a copy of byte string: Resolved
• Merge recent commits from master (7c8160fe) to HDDS-7593: Resolved
• [hsync] Refresh block token immediately if block token expires: Resolved
• OzoneFSInputStream to support ByteBufferPositionedReadable: Resolved
• [hsync] Add a Freon tool to measure client to DataNode round-trip latency: Resolved
• [hsync] Combine WriteData and PutBlock requests into one: Resolved
• [LeaseRecovery] OM shuts down with "SecretKey client must have been initialized already": Resolved
• [hsync] improve block token refresh message: Resolved
• [hsync] Add OpenTracing traces to client side read path: Resolved
• Merge master 97038ef to feature branch HDDS-7593: Resolved
• [hsync] Merge recent commits from master #4: Resolved
• HBase WAL splitting fails due to lease recovery: Resolved
• [hsync] OMKeyCommitRequest should reject if client id doesn't match: Resolved
• [hsync] lease recovery contract test class not substantiated: Resolved
• [hsync] Show deleted hsync keys in ListOpenFile CLI: Resolved
• Merge recent commits from master to HDDS-7593: Resolved
• Show overwritten hsync keys in ListOpenFile CLI: Resolved
• Wrong count is displaying in listOpenFile CLI: Resolved
• [hsync] Output stream lastChunkBuffer should use direct buffer: Resolved
• [hsync] Flush to only wait for majority of DataNodes: Resolved
• Remove unused UserGroupInformation object in DataNode token verifier: Resolved
• Freon tool DN-Echo to support GRPC and Ratis read/write mode: Resolved
• [hsync] Increase default value for hdds.container.ratis.log.appender.queue.num-elements: Resolved
• Intermittent failure in TestLeaseRecovery.testFinalizeBlockFailure: Resolved
• recoverLease should close underlying streams: Resolved
• [hsync] 6th merge from master: Resolved
• Merge master branch 611066a to HDDS-7593 dev branch: Resolved
• [hsync] Parameterize TestBlockOutputStream on ozone.client.stream.putblock.piggybacking: Resolved
• [hsync] Remove block token from Ratis log once verified: Resolved
• [hsync] Investigate why DataNode Echo throughput is so low: Resolved
• [hsync] Client side metrics: Resolved
• [File Lease] OM adds request message handler: Resolved
• [File Lease] Client side lease renewer thread and request message: Resolved
• [File Lease] OM adds FileLeaseManager: Resolved
• [hsync] 7th merge from master: Resolved
• [hsync] Block finalization should also merge last chunk to blockDataTable: Resolved
• [hsync] Adopt Ratis 3.1.0 when it's released: Resolved
• Add a few interesting ContainerStateMachine metrics in CSMMetrics: Resolved
• [hsync] 8th merge from master: Resolved
• [hsync] Checking disk capacity at every write request is expensive for HBase: Resolved
• [hsync] Add a freon tool to benchmark hsync/write concurrency: Resolved
• [hsync] 9th merge from master: Resolved
• [hsync] De-synchronize hsync API: Resolved
• [hsync] Improve BlockOutputStream's BufferPool to support variable buffer allocation from concurrent hsync: Resolved
• [hsync] Replace expensive VolumeUsage.getMinVolumeFreeSpace(): Resolved
• [hsync] Instantiates audit parameter lazily in DataNode dispatch handler: Resolved
• Fix ContainerOpsLatencies metrics: Resolved
• KeyOutputStream flakiness when running write and hsync concurrently: Resolved
• [hsync] Support renaming open files: Resolved
• [hsync] Make Putblock performance acceptable - Tool cleanup, guardrails: Resolved
• Increase hdds.datanode.handler.count: Resolved
• Increase ipc.server.read.threadpool.size: Resolved
• [hsync] Add new OM layout version: Resolved
• [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock: Resolved
• [hsync] Add Ozone Manager protocol version: Resolved
• TestBlockOutputStream.testWriteMoreThanFlushSize is flaky: Resolved
• [hsync] Merge HDDS-7593 feature branch into master: Resolved
• Merge recent commits from master to HDDS-7593: Resolved
• [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test: Resolved
• [hsync] Change XceiverClientRatis.watchForCommit to async: Resolved
• [hsync] Move HBASE_SUPPORT layout upgrade test into its own test: Resolved
• [hsync] Add a client config to limit write concurrency on the same key: Resolved
• [hsync] Revert config default ozone.fs.hsync.enabled to false: Resolved
• [hsync] Block ECKeyOutputStream from calling hsync and hflush: Resolved
• ContainerStateMachine should not crash because of CHUNK_FILE_INCONSISTENCY: Resolved
• Flakiness in KeyOutputStream exception handling: Resolved
• [hsync] Enable PutBlock piggybacking and incremental chunk list by default: Resolved
• Change RatisBlockOutputStream to use HDDS-11174: Resolved
• [hsync] Remove hsync and hflush capability check in ContentGenerator: Resolved
• Use OMLayoutFeature.HBASE_SUPPORT for HSYNC: Resolved
• [hsync] Add upgrade tests: Resolved
• [hsync] Add a config as HBase-related features master switch: Resolved
• [hsync] Remove KeyOutputStreamSemaphore logs: Resolved
• [hsync] Compatibility test: Open
• Support incremental ChunkBuffer checksum calculation: Open
• [hsync] Optimize FilePerBlockStrategy.writeChunk(): Open
• [hsync] Update OpenTelemetry traces in the write path: In Progress
• Make key visible immediately upon create(): Open
• Add hsync and lease recovery Documentation: In Progress
• Add test case to cover HDDS-9930 batch key deletion: Open
• [hsync] Adopt RATIS-1994 to reduce hsync latency: Patch Available
• Any open key rename should ideally be blocked: Open
• Increase DataNode XceiverServerGrpc event loop group size: Open
• [Discuss] Colocate Ozone client and pipeline leader to reduce latency: Open
• [hsync] Freon DN Echo to support --wait-for-commit parameter: Open
• Explore client retry optimizations after write() and hsync() are desynced: Open
• Freon DN Echo should skip writing payload to ratis log: In Progress
• [hsync] Improve test coverage for LeaseRecoveryClientDNHandler: Open
• [hsync] Improve test coverage for XceiverClientRatis.java: Open
• Hsync client-side metrics: Open
• Manage Netty native memory consumption: Open
• Add metrics for OzoneChecksumException: Open