[NUTCH-1297] it is better for fetchItemQueues to select items from greater queues first

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.4
• Component/s: fetcher
• Labels: fetch_queues
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: behnam nikbakht
• Created: 04/Mar/12 06:32
• Updated: 28/Jan/21 14:04
• Resolved: 05/Apr/14 20:14

Description

there is a situation that if we have multiple hosts in fetch, and size of hosts were different, large hosts have a long delay until the getFetchItem() in FetchItemQueues class select a url from them, so we can give them more priority.
for example if we have 10 url from host1 and 1000 url from host2, and have 5 threads, if all threads first selected from host1, we had more delay on fetch rather than a situation that threads first selected from host2, and when host 2 was busy, then selected from host1.

Issue Links

• is related to: NUTCH-1687 Pick queue in Round Robin