In the current folder structure, the separation could look like as follows:
src/test/core - testcases only related to core apis
src/test/hdfs - testcases dependent on hdfs apis. May depend on core apis/tests as well.
src/test/mapred - testcases dependent on mapred apis. May depend on hdfs apis/tests as well.
src/test/hdfs-integration (better name ?)- hdfs tests which depend on mapred for execution. for example benchmark tests NNBench.
There could be separate targets for all the above say - test-core, test-hdfs, test-mapred, test-hdfs-integration in the current build.xml
When the project split, the hdfs subproject need not install mapred. It only needs mapred if wants to run test-hdfs-integration. mapred would have to install hdfs to run its tests as most of the tests run with hdfs.