[CONNECTORS-1392] Add option for Web connector to ignore robots instructions in meta tags

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: ManifoldCF 2.7
• Component/s: Web connector
• Assignee: Markus Schuch
• Reporter: Markus Schuch
• Created: 27/Feb/17 20:48
• Updated: 02/Mar/17 06:02
• Resolved: 02/Mar/17 06:02

Description

The Web connectors already allows to ignore robots.txt by option.

With this ticket, another option is added, to allow the connector to ignore robots instructions in <meta name="robots ... tags.

Proposal (to be discussed)

Add a new option list "Page level robots instructions" to the "Robots" Tab. List entries:

1. Obey meta robots tags (the default)
2. Don't took at meta robots tags

The end user doc needs to be updated.

Google ressources on robot instructions in HTML pages:
[0] https://support.google.com/webmasters/answer/79812?hl=en&ctx=cb&src=cb&cbid=tnnsjq5jcodt&cbrank=4
[1] https://support.google.com/webmasters/answer/96569?hl=en&ctx=cb&src=cb&cbid=-5rmggrfsp2rq&cbrank=3
[2] https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1

Thread on the mailing list
[3] https://www.mail-archive.com/[e-mail]/msg03258.html