I found that with the default settings (the system property "oak.lucene.nonLazyIndex" not set), and a decent number of Lucene indexes, each query that might use a Lucene index takes around 30 ms. With the property set, each query takes only 2 ms or so.
The lazy index download was meant to speed up startup, because only indexes are downloaded that are actually needed. But on my side, I have disabled the setting since then.
I think it makes sense to set change the default value, like it was before. So enable the system property "oak.lucene.nonLazyIndex" by default.