[BEAM-10920] Investigate python hash libraries

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: dependencies, sdk-py-core
• Assignee: Unassigned
• Reporter: Monica Song
• Created: 17/Sep/20 16:43
• Updated: 04/Jun/22 18:57

Description

stats.ApproximateUnique has an optional mmh3 dependency [1] (mmh3 is roughly 9xs faster than md5), but if that repository is problematic for users, we should look into alternatives.

Other options: sklearn.utils.murmurhash3_32

  [1]https://github.com/hajimes/mmh3, https://pypi.org/project/mmh3/2.0/

 

cc: tvalentyn