[NUTCH-1553] Property 'indexer.delete.robots.noindex' not working when using parser-html.

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6
• Fix Version/s: 1.13
• Component/s: indexer, parser
• Assignee: Sebastian Nagel
• Reporter: Alfonso Presa
• Created: 04/Apr/13 14:09
• Updated: 13/Mar/24 14:50
• Resolved: 01/Jul/16 13:17

Description

May be I'm doing something wrong, but it seems to me that NUTCH-1434 patch only works when using tika's parser. When using parser-html, "robots" metatag is only populated if parse-metatags plugin is enabled and it's done with the prefix "metatag.". So parseData.getMeta("robots") returns nothing if not using tika.

I guess the simplest solution would be to provide a fallback in case parseData.getMeta("robots") is null and then get parseData.getMeta("metatag.robots") in that case.

Also dependency of this property with parse-metadata plugin when using parse-html would be something interesting to document somewhere... (nutch-default.xml?)

Thanks!

Issue Links

• breaks: NUTCH-1434 Indexer to delete robots noIndex