[HDDS-7825] Difficult to debug why a node is excluded

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: EC Client
• Labels: pull-request-available
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 23/Jan/23 18:47
• Updated: 30/May/23 18:39
• Resolved: 24/Jan/23 11:38

Description

When the client adds a node to the ExcludedNodes list due to an exception, the exception is hidden. I've found no logging to help determine why a node is added to the ExcludedNodes list, which makes it nearly impossible to determine if some configuration needs to be changed.

In my case it turned out I was running into port exhaustion because lots of sockets were being opened/closed quickly, leaving many in TIME_WAIT, and I was on an older kernel with tcp_tw_reuse=0. One thing that helped determine that was the cause was doing a custom build of Ozone where I added logging when a node is excluded to see what exception caused it. Turns out that exception wasn't related to the target node.

Issue Links

• links to: GitHub Pull Request #4201