|
[
Permlink
| « Hide
]
Stefan Groschupf added a comment - 18/Aug/06 04:50 AM
Since we discussed that nutch need to be more polite we should fix that asap.
I think this issue requires more discussion, especially how it affects the linkdb.
Let's say that page A links to B, but B redirects to C. Issues to discuss:
This is definitely a complex issue. It is also high priority – issues with redirects and duplicates, which URL is chosen, and what happens to the anchor text for the pages involved are causing significant relevance issues.
A few observations: (1) A redirect target is not always the canonical version of a URL. For example, is very common for root-level pages to redirect to an internal home page (some 30% of the root pages in my index do so). However, the root pages have all the anchor text and are truly the canonical, permanent version of the page; the internal redirect target is just the "temporary" homepage, and could change at any time depending on the site implementation. Here are some examples: In this case, we would likely want to mark the internal redirect target as an alias as Andrzej suggests, and automatically transfer any link information to the root page. (2) There may be other cases where we want to alias two pages, either to avoid recrawling them, or to merge anchor text. Suppose we crawl both Right now we will always crawl both of these, and the dedup algorithm will pick one (sadly often the /index.html version due to strange score anomalies), and throw out the anchor text for the other. While we can't safely normalize these two URLs to be the same in advance of seeing the content, once we see that the signatures are the same, we can, and should, merge them so that the index.html version is marked as an alias of the / version, and future crawls simply skip crawling the /index.html version and transfer its link information to the / page. This problem, like the first one, is causing me to lose root-level URLs along with their anchor text, further affecting relevance for navigational queries. In short, I agree with Andrzej that we need a way to mark a URL as an alias of another, to avoid recrawl, and to merge link information. We need to be careful, however, of which URL we pick. It is not always the redirect target that should win. And some of our current concept of "duplicates" should also be subsumed under the new notion of "alias." I'm happy to help out in any way with a fix. I'm just looking at hacking together something in my own environment because the problems are affecting me so severely, but as I'm new-ish to Nutch, what I come up with might not be as elegant or flexible as what others might envision... +1 that the redirect target is not always the "real" URL that we want to keep.
For example, http://www.ibm.com/developerworks/lotus/downloads/toolkits.html It's worth noting that Google, Yahoo! and Microsoft's searches all return lots of links to www-XXX.ibm.com. Just some evidence that this may not be an easy problem to solve.
I don't think there is 100% solution. Mostly because not all respect standards. For example www.imb.com uses 302 status code which by RFC definition - (The requested resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. This response is only cacheable if indicated by a Cache-Control or Expires header field. ). This case is clear. We should use original URL.
But then there is also permanent redirect which SHOULD replace old URL and also update all links pointing to old URL with new one. I also saw some examples of wrong redirections. One of them was my fault to. I use Alias definition with apache server for accepting connections without www subdomain. And then with the page I left link to main page pointing to index.php instead of just /. After a while my domain.si/index.php became more important than www.domain.si (bot points to the same site) So as I see this job is not simple at all. Maybe we need a schema or some sort of flow diagram to indicate what to do in determinant situation. I hope my notes helps a bit because at the moment we really have a lot of unwanted urls in our index. Another small note about this (see
If a page (e.g. http://boutell.com Wait, looks like maybe change 490607 (fix for
I believe the patch in
The other parts though (correct treatment of inlink information, and selection of representative pages for chains of redirects) is not addressed yet. I have a local fix for this problem (partly Paul Gauthier's work, partly mine) that I have been testing for some time. It's a little bit of a hack, but it's much better than just indexing the redirect target (which is the wrong behavior in many instances; see comments earlier).
The fix is to index both instances of the page, both the source and the target, making sure that the outlinks from the target page are only assigned to the target page. This way, in the (frequent) case that the redirect source is the canonical version of the page, with more anchor text, it will show up for searches. The fix seems to work pretty well, and solves a significant percentage of Nutch's "missing home pages" problem without using much extra space in the index. If it sounds useful to anyone, I'm happy to contribute it back. Doug Doug,
Let's see what you got. I'd be happy to take a look at it. Cheers, This i partially fixed so that page status is consistent. LinkDb related changes will be implemented later.
Actually, the problem in the issue description is solved now. I'm closing this one, and the remaining functionality should be tracked as an enhancement in a separate issue.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Major
pages that serverside forwards will be refetched every time
