[HADOOP-17052] NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.0, 2.9.2, 3.2.1, 3.1.3
• Fix Version/s: 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1, 3.4.0
• Component/s: net
• Target Version/s: 3.3.0, 2.9.3, 3.1.4, 3.2.2, 2.10.1
• Hadoop Flags: Reviewed
• Assignee: Dhiraj Hegde
• Reporter: Dhiraj Hegde
• Created: 26/May/20 06:21
• Updated: 01/Jun/20 18:19
• Resolved: 01/Jun/20 18:19

Description

Hadoop components are increasingly being deployed on VMs and containers. One aspect of this environment is that DNS is dynamic. Hostname records get modified (or deleted/recreated) as a container in Kubernetes (or even VM) is being created/recreated. In such dynamic environments, the initial DNS resolution request might return resolution failure briefly as DNS client doesn't always get the latest records. This has been observed in Kubernetes in particular. In such cases NetUtils.connect() appears to throw java.nio.channels.UnresolvedAddressException.  In much of Hadoop code (like DFSInputStream and DFSOutputStream), the code is designed to retry IOException. However, since UnresolvedAddressException is not child of IOException, no retry happens and the code aborts immediately. It is much better if NetUtils.connect() throws java.net.UnknownHostException as that is derived from IOException and the code will treat this as a retry-able error.

Issue Links

• links to: GitHub PR #2036