DistinctPrefixFilter seeks to a smaller key than the current key (which causes an infinite loop in HBase 1.4 and seeks to every row in other HBase versions). This happens when:
- Last column of distinct is descending. We currently always add a 0x01 byte, but since the separator byte if 0xFF when descending, the seek key is too small.
- Last column value is null. In this case, instead of adding a 0x01 byte, we need to increment in-place the null value of the last distinct column.
This was discovered due to OrderByIT.testOrderByReverseOptimizationWithNUllsLastBug3491 hanging in master.