[HDDS-1163] Basic framework for Ozone Data Scrubber

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.1
• Component/s: Ozone Datanode
• Target Version/s: 0.5.0
• Assignee: Supratim Deka
• Reporter: Supratim Deka
• Created: 22/Feb/19 06:39
• Updated: 05/Aug/19 14:45
• Resolved: 12/Mar/19 19:33

Description

Included in the scope:
1. Background scanner thread to iterate over container set and dispatch check tasks for individual containers
2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for example 1 container/s)
3. Check disk layout of Container - basic check for integrity of the directory hierarchy inside the container, include chunk directory and metadata directories
4. Check container file - basic sanity checks for the container metafile
5. Check Block Database - iterate over entries in the container block database and check for the existence and accessibility of the chunks for each block.

Not in scope (will be done as separate subtasks):
1. Dynamic scheduling/pacing of background scan based on system load and available resources.
2. Detection and handling of orphan chunks
3. Checksum verification for Chunks
4. Corruption handling - reporting (to SCM) and subsequent handling of any corruption detected by the scanner. The current subtask will simply log any corruption which is detected.