The submit-lock in nimbus seems to protect some very large and slow sections of code. As more and more topologies are submitted scheduling can take longer and longer to complete making submitting a topology take increasingly longer. But most of scheduling does not need to be protected by this lock. Only a small section of the scheduler pulls state from zookeeper that the lock protects elsewhere.
We should split this lock up and protect scheduling separate from protecting StormBase stored in zk.