[HADOOP-9150] Unnecessary DNS resolution attempts for logical URIs - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.0.2-alpha, 3.0.0-alpha1
Fix Version/s: 2.1.0-beta
Component/s: fs/s3, ha, performance, viewfs
Labels:
None

Target Version/s:

2.0.3-alpha
Hadoop Flags:

Reviewed

Description

In the FileSystem code, we accidentally try to DNS-resolve the logical name before it is converted to an actual domain name. In some DNS setups, this can cause a big slowdown - eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task wasted a lot of time waiting for slow "not found" responses from DNS.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

tracing-resolver.tgz
17/Dec/12 18:57
2 kB
Todd Lipcon
log.txt
17/Dec/12 18:57
8 kB
Todd Lipcon
hadoop-9150.txt
04/Jan/13 23:35
11 kB
Todd Lipcon
hadoop-9150.txt
07/Jan/13 21:24
11 kB
Todd Lipcon
hadoop-9150.txt
08/Jan/13 21:12
11 kB
Todd Lipcon
hadoop-9150.txt
16/Jan/13 01:03
12 kB
Todd Lipcon
hadoop-9150.txt
16/Jan/13 23:34
12 kB
Todd Lipcon
hadoop-9150.txt
17/Jan/13 06:11
13 kB
Todd Lipcon
hadoop-9150.txt
17/Jan/13 23:16
13 kB
Todd Lipcon
hadoop-9150.txt
23/Jan/13 20:13
18 kB
Todd Lipcon
hadoop-9150.txt
24/Jan/13 00:02
19 kB
Todd Lipcon
hadoop-9150.txt
26/Mar/13 23:33
14 kB
Todd Lipcon

Issue Links

is related to

HDFS-6789 TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7

Closed

Activity

People

Assignee:: Todd Lipcon

Reporter:: Todd Lipcon

Votes:: 0 Vote for this issue

Watchers:: 15 Start watching this issue

Dates

Created:: 17/Dec/12 18:54

Updated:: 12/May/16 18:21

Resolved:: 28/Mar/13 20:59