re 1) we should fix the test pkging, also split out units vs func etc... JUnit4 has "category" annotations. We should use these to define things like long/short running tests, etc... Not sure if we want to separate diff test types into diff directories (unit/func/perf/etc...) or use cats, or some combination but we need to think about that as well. src/java/test vs say src/java/units & src/java/func etc... Konstantin ideas for best practices here?
My recent experience with having a physical taxonomy of tests (unit/func/perf) in Hadoop seems to be pretty positive. I.e. it makes build layout simpler. Tests categorization is more clear this way, too. JUnit Categories are rather primitive in compare with Spring or TestNG ones (or at least it used to be like that last time I've checked).
re 4) maybe create a jira for it and link? I'd like to see perf tests run as part of some sort of "nightly" build. Not sure if apache hudson can do this or not though... Konstantin any suggestions on this? What does MR/HDFS do for this?
For a project with a relatively small test base it might be possible to execute everything every, say, night. For MR/HDFS we don't run any performance tests on Apache infrastructure. However, whatever JUnit based tests we have are all ran a couple times a day.
What is important to mention, is that performance/stress results need to be properly trended/analyzed. Otherwise, it doesn't make much sense to bother with these types of tests.
Having subtasked JIRAs for stress and performance tests are good idea, because these are pretty different and are better be tracked separately.
5) there's been some interest to move to maven, would this help/hurt us and should we do that beore/after doing this? is there a "best" order for doing these changes?
It seems to be that client/server separation and build re-org seem like a logical first steps. Mockito + unit tests might come next. Stress/performance are ok to be added lately, because these might require some additional infrastructure/framework efforts.