The compat issue I'm thinking of is if an application is depending on the exact Guava 11 being exposed on the Hadoop classpath. If we change the Guava version, this application will break when it tries to run against the new version of Hadoop. Providing our own versions of removed functions isn't bulletproof since an app could be depending on the behavior from our exact version of Guava. If a function's behavior is different in different versions of Guava, we can't expose all of them at once. I also don't think the Hadoop project wants to be in the business of maintaining our own "compatible" version of Guava.
Basically we're hamstrung since we leak our dependencies. Old apps can depend on our ancient deps, so we have to keep them for compat even though it's quite painful. For Hadoop 3, one option is simply shading all our deps, but that's still under discussion.
For Hadoop 2, there are some possible mitigations. YARN has some classloader and classpath mangling features which can help, but that doesn't help if you're running outside of a YARN container. You can also try just putting your newer deps at the front of the classpath and hoping for the best, but that's unsupported/untested, and is probably something you've already wrestled with.