(I edited your comment to correct your reference to me, not a similarly named person)
I think the surgery makes sense. It adds a useful feature. The approach leverages the existing underlying BytesRefHash which may not be as optimal as some sort of LongHash or similar but whatever – progress not perfection. Should someone care, such improvements could be made later.
I admit I didn't look at the details of your tests; that would take much more time. I was mostly curious about the implementation side and of the lambdas you made reference to.