[DATAFU-46] Hash UDFs should return zero-padded strings of uniform length even when leading bits are zero

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Assignee: Flip Kromer
• Reporter: Matthew Hayes
• Created: 18/May/14 18:37
• Updated: 02/Nov/15 17:58
• Resolved: 19/May/14 01:38

Description

Reported by Philip Kromer here:

https://github.com/linkedin/datafu/issues/93

Details reported there by Philip:

---------------------

The Hash UDFs in 'hex' mode currently do not return always the same-length string, because BigInteger.toString() omits leading zeros. So amidst a stream of 94% strings the same length, 1/16th are shorter by one or more characters, 1/256th by two or more, and in the unlikely case that an MD5 hash's value was 124 bits of zeros and 4 bits of ones it would return the one-character-long string 'f'.

This is surprising behavior, and a trap for those practicing the frequent trick of generating a hash and chopping off just the number of bits you need:

-- returns one-fifteenth, not one-sixteenth, of the input.
sampled_lines = FILTER(FOREACH lines GENERATE MD5(val) AS digest, val) BY (STARTSWITH(digest, 'f'));

Issue Links

• is depended upon by: DATAFU-47 UDF for Murmur3 (and other) Hash functions