Observers will disconnect when the voting peers perform a leader election and reconnect after. The delay zookeeper.observer.reconnectDelayMs was added to insulate the voting peers from the observers returning. With a large number of peers and the observerMaster feature active, this delay is mostly detrimental as it means that the observer is more likely to get hung up on connecting to a bad (down/corrupt) peer and it would be better off switching to a new one quickly.
To retain the protective virtue of the delay, it makes sense to add a delay that after all observer master's in the list have been tried before iterating through the list again. In the case where observer master's are not active, this degenerates to a delay between connection attempts on the leader.