No... it's much more difficult to account for all the mud in the water by running different options in the same JVM run.
It is well known that the first run will be paying the penalty anyway. That is obvious from results I posted last. The non-cache run was 20% slower in the dry run and the next run with same set of params yield only a 5% advantage.
I ran the same multiple times and I never saw a case where the cache was slower.
As we know that caching has better memory efficiency and the performance is marginally better, it is worth investigating what is the real performance before we ruling out this solution
It's simplest to just not put mud in the water in the first place
I don't think you approach of fresh JVM is ideal because in reality we have a JVM that is warmed . According to me the better approach is to run this several times and look at the consistency of the numbers across different runs than running on a cold VM all the time. And my runs really show that caching consistently outperformed non caching (This is not into taking the GC costs into consideration at all)