[CONNECTORS-1657] Web connector - Handle sitemap instruction in robot.txt

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: ManifoldCF 2.17
• Component/s: Web connector
• Assignee: Unassigned
• Reporter: Julien Massiera
• Created: 22/Oct/20 13:58
• Updated: 07/Jul/21 14:41

Description

Currently the web connector does not understand when the robot.txt file points a sitemap. As an example, for the site https://www.persee.fr, in the simple history one can find the following error:

Unknown robots.txt line: 'Sitemap: https://www.persee.fr/sitemap.xml'