Agreed it's risky to blindly pull in all test scope JARs, so maybe it needs to be an explicit list of jars to include.
are you suggesting to create a new libtest/ dir? Wouldn't make more sense to have a tools module for this? this seems like a tool usecase.
I'm not sure it's best to clutter the tools directory with the unit test jars. They're not really related, and it can lead to some of the same problems we're trying to avoid when someone runs tools. IMHO the test stuff shouldn't be in the classpath unless you're running tests, and tools aren't tests.
So yes, I'm proposing a separate place to store the tests and their dependencies. This implies the user needs to modify the classpath to run the tests, and we could make this easier by providing a "hadoop tests-classpath" tool or something similar to aid them in doing so.