Major Description
LatLonPoint needed two-phase intersection initially because of big inefficiencies, but as of LUCENE-7239 all of its query operations: newBoxQuery(), newDistanceQuery(), newPolygonQuery() and nearest() only need the points datastructure (BKD).
If you want to do newDistanceSort() then you need docvalues for that, but I think it should be moved to a separate field: e.g. docvalues is optional just like any other field in lucene. We can add other methods that make sense to that new docvalues field (e.g. facet by distance/region, expressions support, whatever). It is really disjoint from the core query support: and also currently has a heavyish cost of ~64-bits per value in space.