[HDDS-8727] Defer non-critical partial EC reconstruction

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Component/s: SCM
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Attila Doroszlai
• Reporter: Varsha Ravi
• Created: 30/May/23 09:53
• Updated: 01/Jun/23 18:50
• Resolved: 01/Jun/23 18:50

Description

With the new replication manager and throttling feature, if there are 2 missing replicas in a EC container, the expectation is to create a single EC reconstruction command and send it to a target DN which will act as coordinator. But there are multiple reconstruction commands being sent to each individual target DN.

 

2023-05-30 07:14:00,537 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected. bd768c8b-1f30-41a5-b989-40fa5ba07da7
2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size 19 for DN bd768c8b-1f30-41a5-b989-40fa5ba07da7
2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node: /default/bd768c8b-1f30-41a5-b989-40fa5ba07da7

2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected. 116c39e4-4e96-4158-b609-3498d1175abe
2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size 21 for DN 116c39e4-4e96-4158-b609-3498d1175abe
2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node: /default/116c39e4-4e96-4158-b609-3498d1175abe

2023-05-30 07:17:30,597 [Under Replicated Processor] INFO org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending command [reconstructECContainersCommand: containerID: 54, replicationConfig: EC{rs-3-2-1024k}, sources: [d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4, 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988], missingIndexes: [1]] for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 with datanode deadline 1685432820597 and scm deadline 1685432850597
2023-05-30 07:17:30,597 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler: Exception while creating the replication or reconstruction commands for container #54.
2023-05-30 07:17:30,597 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
2023-05-30 07:21:00,646 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler: Exception while creating the replication or reconstruction commands for container #54.
2023-05-30 07:21:00,646 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
2023-05-30 07:21:30,655 [Under Replicated Processor] INFO org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending command [reconstructECContainersCommand: containerID: 54, replicationConfig: EC{rs-3-2-1024k}, sources: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 replicaIndex: 1, d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4, 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets: [4ee90fec-7046-4c07-bd1e-8f6e3ae20520], missingIndexes: [3]] for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to 4ee90fec-7046-4c07-bd1e-8f6e3ae20520 with datanode deadline 1685433060655 and scm deadline 1685433090655

Issue Links

• links to: GitHub Pull Request #4810