[DTACLOUD-440] We need a robots.txt

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Server
• Assignee: Marios Andreou
• Reporter: David Lutterkort
• Created: 26/Jan/13 19:58
• Updated: 29/Jan/13 16:37
• Resolved: 29/Jan/13 16:37

Description

The public endpoints get crawled leading to lots of 403's. We should have a /robots.txt that tells crawlers not to bother looking any further.