[NUTCH-882] Design a Host table in GORA

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: nutchgora
• Fix Version/s: nutchgora
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Julien Nioche
• Created: 11/Aug/10 11:26
• Updated: 27/Apr/12 05:37
• Resolved: 26/Apr/12 09:19

Description

Having a separate GORA table for storing information about hosts (and domains?) would be very useful for :

• customising the behaviour of the fetching on a host basis e.g. number of threads, min time between threads etc...
• storing stats
• keeping metadata and possibly propagate them to the webpages
• keeping a copy of the robots.txt and possibly use that later to filter the webtable
• store sitemaps files and update the webtable accordingly

I'll try to come up with a GORA schema for such a host table but any comments are of course already welcome

Issue Links

• depends upon: GORA-105 DataStoreFactory does not properly support multiple stores
• relates to: NUTCH-628 Host database to keep track of host-level information
• supercedes: NUTCH-1290 crawlId not supported by all Tools