[TS-852] hostdb and dns system hangup

• Type: Bug
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0, sometime
• Component/s: Core, DNS
• Labels: dns hostdb
• Assignee: Thomas Jackson
• Reporter: Zhao Yongming
• Created: 22/Jun/11 07:54
• Updated: 08/Apr/16 18:34

Description

in my testing, all request that need go OS, fails with 30s timeout, and the slow query shows data from dns missed:

[Jun 22 15:33:47.183] Server {1106880832} ERROR: [8122411] Slow Request: url: http://download.im.alisoft.com/aliim/AliIM6/update/packages//2011-5-4-10-10-40/0/modulesproxy/8203.xml.wwzip status: 0 unique id:  bytes: 0 fd: 0 client state: 6 server state: 0 ua_begin: 0.000 ua_read_header_done: 0.000 cache_open_read_begin: 0.000 cache_open_read_end: 0.000 dns_lookup_begin: 0.000 dns_lookup_end: -1.000 server_connect: -1.000 server_first_read: -1.000 server_read_header_done: -1.000 server_close: -1.000 ua_close: 30.667 sm_finish: 30.667
[Jun 22 15:33:47.220] Server {1099663680} ERROR: [8122422] Slow Request: url: http://img.uu1001.cn/materials/original/2010-11-22/22-48/a302876929a9c40a8272ac439a16ad25e74edf71.png status: 0 unique id:  bytes: 0 fd: 0 client state: 6 server state: 0 ua_begin: 0.000 ua_read_header_done: 0.000 cache_open_read_begin: 0.000 cache_open_read_end: 0.000 dns_lookup_begin: 0.000 dns_lookup_end: -1.000 server_connect: -1.000 server_first_read: -1.000 server_read_header_done: -1.000 server_close: -1.000 ua_close: 30.318 sm_finish: 30.318
[Jun 22 15:33:47.441] Server {1098611008} ERROR: [8122421] Slow Request: url: http://img02.taobaocdn.com/bao/uploaded/i2/T1mp8QXopdXXblNIZ6_061203.jpg_b.jpg status: 0 unique id:  bytes: 0 fd: 0 client state: 6 server state: 0 ua_begin: 0.000 ua_read_header_done: 0.000 cache_open_read_begin: 0.000 cache_open_read_end: 0.000 dns_lookup_begin: 0.000 dns_lookup_end: -1.000 server_connect: -1.000 server_first_read: -1.000 server_read_header_done: -1.000 server_close: -1.000 ua_close: 30.597 sm_finish: 30.597
[Jun 22 15:33:47.441] Server {1098611008} ERROR: [8122409] Slow Request: url: http://img04.taobaocdn.com/tps/i4/T1EM9gXlttXXXXXXXX-440-135.jpg status: 0 unique id:  bytes: 0 fd: 0 client state: 6 server state: 0 ua_begin: 0.000 ua_read_header_done: 0.000 cache_open_read_begin: 0.000 cache_open_read_end: 0.000 dns_lookup_begin: 0.000 dns_lookup_end: -1.000 server_connect: -1.000 server_first_read: -1.000 server_read_header_done: -1.000 server_close: -1.000 ua_close: 30.948 sm_finish: 30.948

all http SM show from

{http}

in DNS_LOOKUP

and traffic.out show nothing with error, when I enable debug on hostdb.|dns.:

[Jun 22 15:27:42.391] Server {1108986176} DEBUG: (hostdb) probe img03.taobaocdn.com DB357D60B8247DC7 1 [ignore_timeout = 0]
[Jun 22 15:27:42.391] Server {1108986176} DEBUG: (hostdb) timeout 64204 1308663404 300
[Jun 22 15:27:42.391] Server {1108986176} DEBUG: (hostdb) delaying force 0 answer for img03.taobaocdn.com [timeout 0]
[Jun 22 15:27:42.411] Server {1108986176} DEBUG: (hostdb) probe img03.taobaocdn.com DB357D60B8247DC7 1 [ignore_timeout = 0]
[Jun 22 15:27:42.411] Server {1108986176} DEBUG: (hostdb) timeout 64204 1308663404 300
[Jun 22 15:27:42.411] Server {1108986176} DEBUG: (hostdb) DNS img03.taobaocdn.com
[Jun 22 15:27:42.411] Server {1108986176} DEBUG: (hostdb) enqueuing additional request
[Jun 22 15:27:42.416] Server {47177168876656} DEBUG: (hostdb) probe 127.0.0.1 E9B7563422C93608 1 [ignore_timeout = 0]
[Jun 22 15:27:42.416] Server {47177168876656} DEBUG: (hostdb) immediate answer for 127.0.0.1
[Jun 22 15:27:42.422] Server {1098611008} DEBUG: (hostdb) probe img08.taobaocdn.com 945198AE9AE37481 1 [ignore_timeout = 0]
[Jun 22 15:27:42.422] Server {1098611008} DEBUG: (hostdb) timeout 64281 1308663327 300
[Jun 22 15:27:42.422] Server {1098611008} DEBUG: (hostdb) delaying force 0 answer for img08.taobaocdn.com [timeout 0]
[Jun 22 15:27:42.441] Server {1098611008} DEBUG: (hostdb) probe img08.taobaocdn.com 945198AE9AE37481 1 [ignore_timeout = 0]
[Jun 22 15:27:42.441] Server {1098611008} DEBUG: (hostdb) timeout 64281 1308663327 300
[Jun 22 15:27:42.441] Server {1098611008} DEBUG: (hostdb) DNS img08.taobaocdn.com
[Jun 22 15:27:42.441] Server {1098611008} DEBUG: (hostdb) enqueuing additional request
[Jun 22 15:27:42.470] Server {1099663680} DEBUG: (hostdb) probe 127.0.0.1 E9B7563422C93608 1 [ignore_timeout = 0]
[Jun 22 15:27:42.470] Server {1099663680} DEBUG: (hostdb) immediate answer for 127.0.0.1
[Jun 22 15:27:42.490] Server {1106880832} DEBUG: (hostdb) probe 127.0.0.1 E9B7563422C93608 1 [ignore_timeout = 0]
[Jun 22 15:27:42.490] Server {1106880832} DEBUG: (hostdb) immediate answer for 127.0.0.1
[Jun 22 15:27:42.594] Server {1108986176} DEBUG: (hostdb) probe 127.0.0.1 E9B7563422C93608 1 [ignore_timeout = 0]
[Jun 22 15:27:42.594] Server {1108986176} DEBUG: (hostdb) immediate answer for 127.0.0.1
[Jun 22 15:27:42.818] Server {1108986176} DEBUG: (hostdb) probe 127.0.0.1 E9B7563422C93608 1 [ignore_timeout = 0]
[Jun 22 15:27:42.818] Server {1108986176} DEBUG: (hostdb) immediate answer for 127.0.0.1
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) dequeuing additional request
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) probe img.taobao.com 55D857190294CA66 1 [ignore_timeout = 0]
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) timeout 64134 1308663475 300
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) DNS img.taobao.com
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (dns) received query img.taobao.com type = 1, timeout = 0
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (dns) enqueing query img.taobao.com
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (dns) adding first to collapsing queue
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) probe img.taobao.com 55D857190294CA66 1 [ignore_timeout = 0]
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) timeout 64134 1308663475 300
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) DNS img.taobao.com
[Jun 22 15:27:43.014] Server {1099663680} DEBUG: (hostdb) enqueuing additional request

this is a critical fail for me!

Issue Links

• relates to: TS-2017 DNS lookups stop for a period after ATS receives a response it was not expecting