[HBASE-18410] FilterList Improvement.

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Filters
• Hadoop Flags: Incompatible change, Reviewed

Release Note

In this task, we fixed all existing bugs in FilterList, and did the code refactor which ensured interface compatibility .

The primary bug fixes are :
1. For sub-filter in FilterList with MUST_PASS_ONE, if previous filterKeyValue() of sub-filter returns NEXT_COL, we cannot make sure that the next cell will be the first cell in next column, because FilterList choose the minimal forward step among sub-filters, and it may return a SKIP. so here we add an extra check to ensure that the next cell will match preivous return code for sub-filters.
2. Previous logic about transforming cell of FilterList is incorrect, we should set the previous transform result (rather than the given cell in question) as the initial vaule of transform cell before call filterKeyValue() of FilterList.
3. Handle the ReturnCodes which the previous code did not handle.

About code refactor, we divided the FilterList into two separated sub-classes: FilterListWithOR and FilterListWithAND, The FilterListWithOR has been optimised to choose the next minimal step to seek cell rather than SKIP cell one by one, and the FilterListWithAND has been optimised to choose the next maximal key to seek among sub-filters in filter list. All in all, The code in FilterList is clean and easier to follow now.

Note that ReturnCode NEXT_ROW has been redefined as skipping to next row in current family, not to next row in all family. it’s more reasonable, because ReturnCode is a concept in store level, not in region level.

Another bug that needs attention is: filterAllRemaining() in FilterList with MUST_PASS_ONE will now return false if the filter list is empty whereas earlier it used to return true for Operator.MUST_PASS_ONE. it's more reasonable now.

• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 19/Jul/17 06:53
• Updated: 30/Jun/21 15:39
• Resolved: 25/Oct/17 12:42

Description

FilterList.java is complex now, and we have found some improvements for it. So create an issue to address it.

Issue Links

• is related to: HBASE-17678 FilterList with MUST_PASS_ONE may lead to redundant cells returned
• is related to: HBASE-18879 HBase FilterList cause KeyOnlyFilter not work
• is related to: HBASE-19640 Change in filterAllRemaining() impl for FilterListWithOR
• is related to: HBASE-18993 Backport patches in HBASE-18410 to branch-1.x branches.
• relates to: HBASE-20565 ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
• relates to: HBASE-18904 Missing break in NEXT_ROW case of FilterList#mergeReturnCodeForOrOperator()
• relates to: HBASE-19252 Move the transform logic of FilterList into transformCell() method to avoid extra ref to question cell

Sub-Tasks

• add test that confirms 2 FamilyFilters in a FilterList using MUST_PASS_ONE operator will return results that match either of the FamilyFilters and revert as needed to make it pass.: Resolved
• Fix incorrect logic in FilterList.filterKeyValue: Resolved
• Dividing FiterList into two separate sub-classes: FilterListWithOR , FilterListWithAND: Resolved
• FilterList with MUST_PASS_ALL can be optimized to choose the highest key among filters in filter list to seek: Resolved
• FilterLIst with MUST_PASS_ONE can be optimized to choose the mininal step to seek rather than SKIP cell one by one: Resolved
• FilterList with multiple FamilyFilters concatenated by OR does not work.: Resolved
• FilterList with MUST_PASS_ONE may lead to redundant cells returned: Resolved
• Reenable test of filterlist using MUST_PASS_ONE and two familyfilters: Resolved
• HBase FilterList cause KeyOnlyFilter not work: Resolved
• Fix other code review comments about FilterList Improvement: Resolved