[NUTCH-2124] redirect following same link again and again , max redirect exceed and went db_gone

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.11
• Component/s: fetcher
• Labels: db_gone fetcher redirect
• Patch Info: Patch Available
• Flags: Important
• Assignee: Sebastian Nagel
• Reporter: Yogendra Kumar Soni
• Created: 28/Sep/15 14:03
• Updated: 13/Mar/24 14:51
• Resolved: 07/Oct/15 19:03

Description

Hello, followredirect is not working in trunk. please see the below log.

Fetcher: throughput threshold retries: 5
fetcher.maxNum.threads can't be < than 50 : using 50 instead
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1

fetching http://www.wikipedia.com/wiki/URL_redirection (queue crawl delay=5000ms)
fetching http://www.wikipedia.com/wiki/URL_redirection (queue crawl delay=5000ms)
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=2
fetching http://www.wikipedia.com/wiki/URL_redirection (queue crawl delay=5000ms)
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=2
fetching http://www.wikipedia.com/wiki/URL_redirection (queue crawl delay=5000ms)
fetching http://www.wikipedia.com/wiki/URL_redirection (queue crawl delay=5000ms)
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=2
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=2
- redirect count exceeded http://www.wikipedia.com/wiki/URL_redirection
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=2
-activeThreads=0
Fetcher: finished at 2015-09-28 19:32:05, elapsed: 00:00:09
Parsing : 20150928193153