[HDDS-8161] Atomic container filesystem operations

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Ozone Datanode
• Assignee: Unassigned
• Reporter: Ethan Rose
• Created: 14/Mar/23 22:51
• Updated: 23/Mar/23 18:36

Description

Parent Jira to track tasks related to using filesystem mv to make changes to containers atomically without polluting the main container directory that datanodes require to be in a consistent state. The proposal is one tmp directory per volume located at hdds/<clusterID>/tmp which would hold files and directories under construction. They would then be moved to the working directory on completion using mv. The tmp directory would have subdirectories for each task, for example hdds/<clusterID>/tmp/container-copy, hdds/<clusterID>/tmp/container-create etc. These directories could be cleaned out on restart or as needed.

Issue Links

• is a child of: HDDS-6548 HDDS Stability Fixes and Improvements
• is a parent of: HDDS-6449 Failed container delete can leave artifacts on disk
• relates to: HDDS-6302 /tmp/container-copy directory left over after volume/bucket/key deletion

Sub-Tasks

• Spread container-copy directory on different volumes: Resolved
• Failed container delete can leave artifacts on disk: Resolved
• Refactor volume level tmp directory for generic usage: Resolved
• Cleanup of failed container delete may remove datanode RocksDB entries of active container: Resolved
• Reconcile volume tmp directories for different features: Open
• Move block files to container directory after the first put block: Open