[NUTCH-1062] Migrate BasicURLNormalizer from Apache ORO to java.util.regex

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10, 2.3.1
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 19/Jul/11 12:10
• Updated: 13/Mar/24 14:50
• Resolved: 22/Apr/15 21:08

Description

Issue for migration from ORO to j.u.regex. There is a small problem here. I began the migration mostly because of the double slash issue using lookback which was not supported in ORO. This was to prevent the URL schema from being reduced to one slash. The current Basic URL Normalizer has this problem built-in!

        // this pattern tries to find spots like "xx//yy" in the url,
        // which could be replaced by a "/"
        adjacentSlashRule = new Rule();
        adjacentSlashRule.pattern = (Perl5Pattern)      
          compiler.compile("/{2,}", Perl5Compiler.READ_ONLY_MASK);     
        adjacentSlashRule.substitution = new Perl5Substitution("/");

But provides the wrong solution as it touches the schema as well. What to do? Migrate to j.u.regex and keep this `feature` intact?

edit: reading more it looks like it is being fixed at a later stage. A slash is added for URI schema's http & ftp.

Issue Links

• blocks: NUTCH-1014 Migrate from Apache ORO to java.util.regex