[#NUTCH-411] Parse ignores meta refresh redirection

Nutch

Parse ignores meta refresh redirection

Created: 30/Nov/06 02:35 PM Updated: 10/Apr/09 12:29 PM

Return to search

Component/s:

None

Affects Version/s:

0.9.0

Fix Version/s:

1.0.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	parse-redirect.patch	2006-11-30 02:53 PM	Doğacan Güney	1 kB

Issue Links:

Reference

This issue relates to:

~~NUTCH-572~~ Scoring and redirected Urls

Resolution Date:

08/Nov/07 03:03 PM

Description

« Hide

If fetching and parsing are run as seperate jobs, then redirection coming from meta refresh tag (i.e. <meta http-equiv="refresh" content="0;url=foo/">) is ignored, resulting in the loss of that ("foo/") url.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Doğacan Güney added a comment - 30/Nov/06 02:52 PM

My not-necessarily-correct patch for this. We add the new url as a newly discovered url (so it gets initialScore), which is different from what happens if we parse in fetcher.

I believe that in the long term, nutch should associate source url with the redirected url. But this patch (or a more correct version of this) can be applied so that we do not lose urls in the short term.

[ Permlink | « Hide ]

Doğacan Güney added a comment - 08/Nov/07 03:03 PM

This is fixed as part of ~~NUTCH-547~~.