[HDDS-3214] Unhealthy datanodes repeatedly participate in pipeline creation

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Component/s: SCM
• Labels: TriagePending fault_injection
• Assignee: Ethan Rose
• Reporter: Nilotpal Nandi
• Created: 16/Mar/20 08:18
• Updated: 21/Jan/24 15:44
• Resolved: 08/Aug/23 19:42

Description

steps taken :

1) Mounted noise injection FUSE on all datanodes

2) Selected 1 datanode from each open pipeline (factor=3)

3) Injected WRITE FAILURE noise with error code - ENOENT on "hdds.datanode.dir" path of list of datanodes selected in step 2)

4) start PUT key operation of size  32 MB.

 

Observation :

----------------

1. Commit failed, pipelines were moved to exclusion list.
2. Client retries , new pipeline is created with same set of datanodes. Container creation fails as WRITE  FAILURE injection present.
3. Pipeline is closed and the process is repeated for "ozone.client.max.retries" retries.

Everytime, same set of datanodes are selected for pipeline creation which include 1 unhealthy datanode. 

Expectation - pipeline should have been created by selecting 3 healthy  datanodes available.

 

cc - ljain

 

Issue Links

• is related to: HDDS-7365 Integrate container and volume scanners