the loss of features in the UI (they are sticking to 2.4.1's "legacy mode")
Which feature is lost? Please feel free to file a jira to track this.
"tossed over the fence" WebHDFS authentication that we had to rework to even make security function if one was using a custom plug-in.
This is an unfair statement. The JSP UI has a series of security issues (e.g., XSS) because of its architecture. The new UI is much more secure as these issues are addressed systematically. Please familiar yourself with relevant jiras.
The switch took a lot of effort to even get to stable and it's still not what I would consider feature complete compared to the old API.
The same argument applies to any components. For example, how long does Yetus take to stabilize? Do you consider Yetus feature complete compared to the test-patch.sh? Yetus is a good improvement in terms of usability and maintainability. It also brings in more people in the community. So does the HTML5 UI.
Going back to the original topic, are there any specific reasons to not fixing code that fails to compile under the current JDK? What is a good answer to "why a stable release of Hadoop can't be compiled out-of-the-box?"