[NUTCH-950] Content-Length limit, URL filter and few minor issues

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: nutchgora
• Fix Version/s: nutchgora
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Alexis
• Created: 01/Jan/11 08:06
• Updated: 22/May/13 03:53
• Resolved: 10/Jan/11 10:30

Description

1. crawl command (nutch1.patch)

The class was renamed to Crawler but the references to it were not updated.

2. URL filter (nutch2.patch)

This avoids a NPE on bogus urls which host do not have a suffix.

3. Content-Length limit (nutch3.patch)

This is related to NUTCH-899.
The patch avoids the entire flush operation on the Gora datastore to crash because the MySQL blob limit was exceeded by a few bytes. Both protocol-http and protocol-httpclient plugins were problematic.

4. Ivy configuration (nutch4.patch)

• Change xercesImpl and restlet versions. These 2 version changes are required. The first one currently makes a JUnit test crash, the second one is missing in default Maven repository.

• Add gora-hbase, zookeeper which is an HBase dependency. Add MySQL connector. These jars are necesary to run Gora with HBase or MySQL datastores. (more a suggestion that a requirement here)

• Add com.jcraft/jsch, which is a protocol-sftp plugin dependency.

Sub-Tasks

• Fix crawl command: Closed
• Bugfix for Content-Length limit in http protocols: Closed