[HBASE-5416] Filter on one CF and if a match, then load and return full row (WAS: Improve performance of scans with some kind of filters)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.90.4
• Fix Version/s: 0.94.5, 0.95.0
• Component/s: Filters, Performance, regionserver
• Hadoop Flags: Reviewed

Release Note

New method is added to Filter which allows filter to specify which CF is needed to it's operation.

public boolean isFamilyEssential(byte[] name);

When new row is considered, only data for essential family is loaded and filter applied. And only if filter accepts the row, rest of data is loaded.

This feature is off by default. You can use Scan.setLoadColumnFamiliesOnDemand() to enable it on a per Scan basis. If not indicated for the Scan, boolean value for "hbase.hregion.scan.loadColumnFamiliesOnDemand" would be used (default to false).

• Assignee: Sergey Shelukhin
• Reporter: Max Lapan
• Created: 16/Feb/12 20:26
• Updated: 29/Sep/16 10:09
• Resolved: 14/Jan/13 23:02

Description

When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed.

But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter.

For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed.

Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed.

Issue Links

• blocks: HBASE-7383 create integration test for HBASE-5416 (improving scan performance for certain filters)
• is broken by: HBASE-6499 StoreScanner's QueryMatcher not reset on store update
• is related to: HBASE-8334 Enable essential column family support by default
• relates to: HBASE-16731 Inconsistent results from the Get/Scan if we use the empty FilterList
• relates to: HBASE-7920 Move isFamilyEssential(byte[] name) out of Filter interface in 0.94
• relates to: HBASE-74 [performance] When a get or scan request spans multiple columns, execute the reads in parallel
• relates to: HBASE-16729 Define the behavior of (default) empty FilterList