After some research I found out that this issue is a reoccurring one - which never really has been fixed.
The first one was
TAP5-171 the second TAP5-1044
The problem hasn't changed very much except that deleting the doctype doesn't help anymore since t5.3 as xhtml transitional is used in memory when none is provided (same for the html5 doctype) (source: http://tapestry.apache.org/component-templates.html#ComponentTemplates-TemplateDoctypes).
When using html5 the problem is that invalid html markup is generated, for example:
- a br tag renders as <br clear="none"/> where clear is invalid html5
- an a tag renders as <a shape="rect">... where the shape attribute is invalid html5