Tom, I applied the latest patches for all three sub-projects, cleared my IVY cache, rebuilt all of them. Then, I compared the ivy library dependencies that we are declaring in the sources of the three packages and the libraries in the lib folder of the built packages. In both analyses, I still see some duplicate jars. I think most or all of the duplicates in source are coming from the HDFS Proxy ivy file. I don't know if this is reason why the libraries in the build/lib are ultimately repeating.
For e.g. here's a list of the jars in the ivy properties that have different versions:
commons-logging-api, commons-logging, ivy, jets3t, junit, mockito-all, slf4j-api, slf4j-log4j12
There are quite a few repetitions in the lib folders of the three packages:
I am attaching two text files that are the dumps of the ivy library declarations and the build/lib folder contents for your review.
Please note that I did not run the cluster with these packages - so things are possibly fine with these duplications as well.