[NUTCH-1290] crawlId not supported by all Tools

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: nutchgora
• Fix Version/s: nutchgora
• Component/s: indexer
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Mathijs Homminga
• Created: 28/Feb/12 10:39
• Updated: 26/Apr/12 11:34
• Resolved: 26/Apr/12 11:34

Description

See also: https://issues.apache.org/jira/browse/NUTCH-907

The StorageUtils class exposes a createDataStore method which uses the default schema for a persistent class specified in the Gora configuration.
This method ignores Nutch' storage.schema property and the notion of a crawlId.

Two tools use this method instead of the createWebStore method (which does support the storage.schema property and a crawlId):

o.a.n.indexer.IndexerReducer (IndexerJob)
o.a.n.util.domain.DomainStatistics

I propose that these two start using the createWebStore method and that we make remove the createDataStore method from the StorageUtils.
Also, these two tools should support the crawlId command line parameter.

Issue Links

• is superceded by: NUTCH-882 Design a Host table in GORA