HDFS-330. So that public UIs don't get crawled.
I can provide a patch that includes a simple robots.txt. Another alternative is probably a Filter that provides one automatically for all UIs but I don't have time to do that.
If anyone wants to take over please go ahead.