Thanks Martijn van Groningen, this is nice!
You don't have to box the point dimensionCount/numBytes up from int to Integer: 0 can safely mean "this field has no points".
In addField you return right away if the field had point values, but this is dangerous because the field could also have e.g. doc values: (LatLonPoint just recently started doing this), maybe we can add a test case for that?
Instead of making BytesRefArray.sort fully public, can we make a version that always sorts by natural order public? This class's entire existence scares me so I like to minimize what methods we make public, even for internal usage.
Or maybe MemoryIndex should just use BytesRef, instead of BytesRefArray, and sort that? The vast majority of the time we are looking at a single point value for the fields here? Your visit function could be simplified too: just call the compare only once, and if it crosses, visit all points (with doc and value); if the cell is inside the query, visit with just doc; else, do nothing?