[PHOENIX-4742] DistinctPrefixFilter potentially seeks to lesser key when descending or null value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.14.0, 5.0.0
• Assignee: James R. Taylor
• Reporter: James R. Taylor
• Created: 17/May/18 18:02
• Updated: 26/Jul/18 01:16
• Resolved: 19/May/18 00:19

Description

DistinctPrefixFilter seeks to a smaller key than the current key (which causes an infinite loop in HBase 1.4 and seeks to every row in other HBase versions). This happens when:

1. Last column of distinct is descending. We currently always add a 0x01 byte, but since the separator byte if 0xFF when descending, the seek key is too small.
2. Last column value is null. In this case, instead of adding a 0x01 byte, we need to increment in-place the null value of the last distinct column. 

This was discovered due to OrderByIT.testOrderByReverseOptimizationWithNUllsLastBug3491 hanging in master.