Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
1.19
-
None
-
Nutch 1.19, checked on Windows 10 and Ubuntu. Both have the same issue.
'm trying to crawl a SharePoint intranet using nutch where the URLs are similar to:
https://localnet/something.aspx
The issue is that Nutch is rejecting any url with a single element domain name such as localnet above. "localnet.com" is not rejected, nor is "local.localnet". It almost feels as if there's a chunk of code within Nutch that's unrelated to the filtering mechanisms that rejects URLs outright if they don't have a WWW style format and a WWW-style domain such as .COM
Error message:
Total urls rejected by filters: 1
I've checked and updated all the filter files in the conf directory. Even making then incredibly permissive (effectively "crawl everything") has not helped.
Nutch 1.19, checked on Windows 10 and Ubuntu. Both have the same issue. 'm trying to crawl a SharePoint intranet using nutch where the URLs are similar to: https://localnet/something.aspx The issue is that Nutch is rejecting any url with a single element domain name such as localnet above. "localnet.com" is not rejected, nor is "local.localnet". It almost feels as if there's a chunk of code within Nutch that's unrelated to the filtering mechanisms that rejects URLs outright if they don't have a WWW style format and a WWW-style domain such as .COM Error message: Total urls rejected by filters: 1 I've checked and updated all the filter files in the conf directory. Even making then incredibly permissive (effectively "crawl everything") has not helped.
Description
There appears to be a bug within the core of Nutch that fails to permit any single domain name URLs to be crawled. Example:
https://localnet/something.aspx
The issue is that Nutch is rejecting any url with a single element domain name such as localnet above. "localnet.com" is not rejected, nor is "local.localnet". It almost feels as if there's a chunk of code within Nutch that's unrelated to the filtering mechanisms that rejects URLs outright if they don't have a WWW style format and a WWW-style domain such as .COM
Error message:
Total urls rejected by filters: 1
I've checked and updated all the filter files in the conf directory. Even making then incredibly permissive (effectively "crawl everything") has not helped. Immediately that a dot (.) is added to the domain name it is not rejected - eg blah.localnet.
Attachments
Issue Links
- is superceded by
-
-
- Closed
-