[#NUTCH-273] When a page is redirected, the original url is NOT updated.

Nutch

When a page is redirected, the original url is NOT updated.

Created: 20/May/06 04:23 PM Updated: 28/Dec/06 12:18 AM

Return to search

Component/s:

fetcher

Affects Version/s:

0.8

Fix Version/s:

0.9.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	Fetcher.java-489586.diff	2006-12-22 09:38 AM	Eelco Lempsink	0.6 kB

Environment:

n/a

Issue Links:

Dependants

This issue blocks:

~~NUTCH-353~~ pages that serverside forwards will be refetched every time

Incorporates

This issue is part of:

~~NUTCH-322~~ Fetcher discards ProtocolStatus, doesn't store redirected pages

Reference

This issue is related to:

~~NUTCH-371~~ DeleteDuplicates should remove documents with duplicate URLs

Resolution Date:

28/Dec/06 12:18 AM

Description

« Hide

[Excerpt from maillist, sender: Andrzej Bialecki]
When a page is redirected, the original url is NOT updated - so, CrawlDB will never know that a redirect occured, it won't even know that a fetch occured... This looks like a bug.
In 0.7 this was recorded in the segment, and then it would affect the Page status during updatedb. It should do so 0.8, too...

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#490607	Thu Dec 28 00:03:04 UTC 2006	ab	This patch addresses several issues: * ~~NUTCH-415~~ - Generator should mark selected records in CrawlDb. Due to increased resource consumption this step is optional. Application-level locking has been added to prevent concurrent modification of databases. * ~~NUTCH-416~~ - CrawlDatum status and CrawlDbReducer refactoring. It is now possible to correctly update CrawlDb from multiple segments. Introduce new status codes for temporary and permanent redirection. * ~~NUTCH-322~~ - Fix Fetcher to store redirected pages and to store protocol-level status. This also should fix ~~NUTCH-273~~. * Change default Fetcher behavior not to follow redirects immediately. Instead Fetcher will record redirects as new pages to be added to CrawlDb. This also partially addresses ~~NUTCH-273~~. * Detect and report when Generator creates 0-sized segments. * Fix Injector to preserve already existing CrawlDatum if the seed list being injected also contains such URL. This development was partially supported by SiteSell Inc.
				Files Changed
				MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/CrawlDbMerger.java MODIFY /lucene/nutch/trunk/src/test/org/apache/nutch/crawl/TestCrawlDbMerger.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/tools/compat/CrawlDbConverter.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/MapWritable.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/indexer/Indexer.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/LinkDb.java MODIFY /lucene/nutch/trunk/src/test/org/apache/nutch/crawl/TestInjector.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/Generator.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/Injector.java MODIFY /lucene/nutch/trunk/src/test/org/apache/nutch/crawl/TestGenerator.java MODIFY /lucene/nutch/trunk/CHANGES.txt MODIFY /lucene/nutch/trunk/src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/metadata/Nutch.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/CrawlDbReader.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/Crawl.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/fetcher/Fetcher.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/CrawlDatum.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/CrawlDb.java MODIFY /lucene/nutch/trunk/src/test/org/apache/nutch/fetcher/TestFetcher.java MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/crawl/CrawlDbReducer.java MODIFY /lucene/nutch/trunk/conf/nutch-default.xml MODIFY /lucene/nutch/trunk/src/java/org/apache/nutch/segment/SegmentReader.java ADD /lucene/nutch/trunk/src/java/org/apache/nutch/util/LockUtil.java