Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.21
-
None
Description
I am trying to scan the URL
smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/
Note the userinfo 'hiran', which is used for authentication on the server. (The smb plugin pulls credentials from another configuration file, but this is irrelevant here).
The URL is fetched, parsed, updated in the crawldb and sent to the indexer. So far so good. But the outlinks that are detected are of different quality: some have the userinfo preserved, some are missing that information.
Dumping the segment I can see the below data. Note that some of the outlinks start with smb://hiran@nas.fritz.box, while others start with smb://nas.fritz.box. The impact is that on the next fetch run authentication information is missing and the URLs cannot be fetched further.
Recno:: 0 URL:: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/ CrawlDatum:: Version: 7 Status: 1 (db_unfetched) Fetch time: Tue Oct 29 22:56:58 CET 2024 Modified time: Thu Jan 01 01:00:00 CET 1970 Retries since fetch: 0 Retry interval: 86400 seconds (1 days) Score: 1.0 Signature: null Metadata: _ngt_=1730239026566 Content:: Version: -1 url: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/ base: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/. contentType: text/html metadata: nutch.segment.name=20241029225708 _fst_=33 nutch.crawl.score=1.0 Content: <html><head><title>Index of /Documents/Hiran/Monitoring/</title></head><body><h1>Index of /Documents/Hiran/Monitoring/</h1><pre><a href=".svn/">.svn/ Tue Oct 24 13:32:32 CEST 2017</a> <a href="architektur.dia">architektur.dia Mon Feb 22 21:30:33 CET 2010</a> <a href="architektur.dia%7E">architektur.dia~ Mon Feb 22 21:20:42 CET 2010</a> <a href="architektur.png">architektur.png Mon Feb 22 21:34:27 CET 2010</a> <a href="deployment.dia">deployment.dia Mon Feb 22 22:56:15 CET 2010</a> <a href="deployment.dia%7E">deployment.dia~ Mon Feb 22 22:51:21 CET 2010</a> <a href="deployment.png">deployment.png Mon Feb 22 23:00:34 CET 2010</a> <a href="Monitoring+strategy.odt">Monitoring strategy.odt Fri Aug 01 13:38:04 CEST 2014</a> </pre></body></html> ParseData:: Version: 5 Status: success(1,0) Title: Index of /Documents/Hiran/Monitoring/ Outlinks: 5 outlink: toUrl: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/architektur.dia anchor: architektur.dia Mon Feb 22 21:30:33 CET 2010 outlink: toUrl: smb://nas.fritz.box/Documents/Hiran/Monitoring/architektur.dia~ anchor: architektur.dia~ Mon Feb 22 21:20:42 CET 2010 outlink: toUrl: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/deployment.dia anchor: deployment.dia Mon Feb 22 22:56:15 CET 2010 outlink: toUrl: smb://nas.fritz.box/Documents/Hiran/Monitoring/deployment.dia~ anchor: deployment.dia~ Mon Feb 22 22:51:21 CET 2010 outlink: toUrl: smb://hiran@nas.fritz.box/Documents/Hiran/Monitoring/Monitoring+strategy.odt anchor: Monitoring strategy.odt Fri Aug 01 13:38:04 CEST 2014 Content Metadata: nutch.segment.name = 20241029225708 nutch.content.digest = a794c6675cb2f9e460e7771060ed2dfc _fst_ = 33 nutch.crawl.score = 1.0 Parse Metadata: CharEncodingForConversion = windows-1252 OriginalCharEncoding = windows-1252 language = en CrawlDatum:: Version: 7 Status: 65 (signature) Fetch time: Tue Oct 29 22:57:25 CET 2024 Modified time: Thu Jan 01 01:00:00 CET 1970 Retries since fetch: 0 Retry interval: 0 seconds (0 days) Score: 0.0 Signature: a794c6675cb2f9e460e7771060ed2dfc Metadata: CrawlDatum:: Version: 7 Status: 33 (fetch_success) Fetch time: Tue Oct 29 22:57:17 CET 2024 Modified time: Thu Jan 01 01:00:00 CET 1970 Retries since fetch: 0 Retry interval: 86400 seconds (1 days) Score: 1.0 Signature: null Metadata: _ngt_=1730239026566 _pst_=success(1), lastModified=0 Content-Type=text/html ParseText:: Index of /Documents/Hiran/Monitoring/ Index of /Documents/Hiran/Monitoring/ .svn/ Tue Oct 24 13:32:32 CEST 2017 architektur.dia Mon Feb 22 21:30:33 CET 2010 architektur.dia~ Mon Feb 22 21:20:42 CET 2010 architektur.png Mon Feb 22 21:34:27 CET 2010 deployment.dia Mon Feb 22 22:56:15 CET 2010 deployment.dia~ Mon Feb 22 22:51:21 CET 2010 deployment.png Mon Feb 22 23:00:34 CET 2010 Monitoring strategy.odt Fri Aug 01 13:38:04 CEST 2014
Addendum: It is ok to have only 5 outlinks from a document with 8 anchors. The .svn and the two .png links are ignored. My regex-urlfilter.txt looks like this:
# skip file: ftp: and mailto: urls -^(?:file|ftp|mailto): # skip image and other suffixes we can't yet parse # for a more extensive coverage use the urlfilter-suffix plugin -(?i)\.(?:gif|jpg|png|ico|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$ # skip URLs containing certain characters as probable queries, etc. -[?*!=] # skip URLs with slash-delimited segment that repeats 3+ times, to break loops -.*(/[^/]+)/[^/]+\1/[^/]+\1/ # skip version control internal files -(?i)\.(?:git|svn|cvs)$ # skip recycle bin URLs -(?i)/%23recycle/$ -/\.svn/ -/\.git/ # accept anything else +.
Attachments
Issue Links
- links to