Thanks for the updates, Lars Hofhansl. That's a good trick with the ordering of the filters and it makes sense the the more selective the filter, the less optimization you'd get.
With "reverse" you mean explicit ORDER BY
Yes, like this SELECT prefix1 FROM t GROUP BY prefix1 ORDER BY prefix1 DESC. In this case, the client-side sort is optimized out and a reverse scan will be run instead, so your DistinctPrefixFilter will need to generate the seek hint differently. The way to check on the client is like this (not as I mentioned before): plan.getOrderBy() == OrderBy.REV_ROW_KEY_ORDER_BY
I don't quite follow the RCV example
The col variable is used both for the optimization and to determine how many slots to use during the running of the DistinctPrefixFilter. In the RVC example, the cols would turn out to be just 1, since there'll be single expression (the RVC expression), but it spans two slots. Also, there are other weird cases possible, like this:
SELECT prefix11 FROM t GROUP BY prefix1, TRUNC(prefix1)
In that case, col would be set to 2, but it really should be 1. By letting OrderPreservingTracker track it for you, you'll handle all the weird cases (both to turn off the usage of the filter and to set the number of slots correctly)