[HBASE-17756] We should have better introspection of HFiles

• Type: Brainstorming
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Component/s: HFile
• Assignee: Rushabh Shah Assign to me
• Reporter: Esteban Gutierrez
• Created: 08/Mar/17 00:14
• Updated: 17/Jun/22 17:47
• Resolved: 11/Jun/22 18:37

Description

Stack was suggesting to use DataSketches (https://datasketches.github.io) in order to write additional statistics to the HFiles. This could be used to improve our split decisions, troubleshooting or potentially do other interesting analysis without having to perform full table scans. The statistics could be stored as part of the HFile but we could initially improve the visibility of the data by adding some statistics to HFilePrettyPrinter.

Issue Links

• relates to: HBASE-9113 Expose region statistics on table.jsp
• relates to: HBASE-25641 Improve storeFile.jsp performance on really large files
• relates to: HBASE-12311 Version stats in HFiles?
• links to: GitHub Pull Request #1727
• links to: Performance Test Tool for HFile

Sub-Tasks

• [hbase-operator-tools] Tool to Report on row sizes and column counts: Closed