In fact, I used the 4.0 code +
SOLR-1293 modified code to test lotsofcores solr 4.0 (to be able to handle 50k cores per server with 1000 lives cores max at any time, shared configs, numBuckets, lazy load cores, efficient start cores, automatic start/stop cores, core creation on the fly, efficient fresh start). It's the reason why I have the loadOnStart="false" option.
Anyways, I pull down 5x and without changes, I try to start the server (50K cores with "loadOnStartup=false"), and same issue (very very long time to start ...)
After that, I applied the change (line 590 of CoreContainer.java Apache Solr 5.0 trunck). In this way, the node you are working with is detached from the main document. So, it's a huge speed up for the evaluate method used in readProperties function called for each core, especially in case of lots of cores. And now it takes a few seconds to start the server.
Thanks, I would following
SOLR-4083, because in your specific use case (50k cores), deprecate solr.xml entirely could be a scalability problem (especially fresh start time).