[ACCUMULO-4314] Use statistics to choose better keys for RFile index

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.6, 1.7.2, 1.8.0
• Assignee: Keith Turner
• Reporter: Keith Turner
• Created: 23/May/16 23:15
• Updated: 28/Jun/17 21:15
• Resolved: 31/May/16 23:20

Description

The commit for ACCUMULO-1124 makes two changes :

• Generates shorter keys that may not exist in data to place in RFile index
• Use statistics to make better choices about what keys to place in index. These changes look for keys that are average or below and excludes large keys (keys that are > 3 std dev).

The change to generate shorter keys can not be made in 1.7.X and 1.6.X because it would generate RFiles that may not work properly with older 1.6 and 1.7 versions. However the changes to use statistics to pick better keys could be made in 1.6 and 1.7.

Issue Links

• breaks: ACCUMULO-4669 RFile can create very large blocks when key statistics are not uniform
• relates to: ACCUMULO-1124 optimize index size in RFile