[SPARK-3847] Enum.hashCode is only consistent within the same JVM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.1.0
• Component/s: Spark Core
• Labels: bulk-closed enum

Environment

Oracle JDK 7u51 64bit on Ubuntu 12.04

• Assignee: Unassigned
• Reporter: Nathan Bijnens
• Created: 08/Oct/14 12:02
• Updated: 29/Jan/20 06:07
• Resolved: 06/Jun/19 13:57

Description

When using java Enum's as key in some operations the results will be very unexpected. The issue is that the Java Enum.hashCode returns the memoryposition, which is different on each JVM.

messages.filter(_.getHeader.getKind == Kind.EVENT).count
>> 503650

val tmp = messages.filter(_.getHeader.getKind == Kind.EVENT)
tmp.map(_.getHeader.getKind).countByValue
>> Map(EVENT -> 1389)

Because it's actually a JVM issue we either should reject with an error enums as key or implement a workaround.

A good writeup of the issue can be found here (and a workaround):
http://dev.bizo.com/2014/02/beware-enums-in-spark.html

Somewhat more on the hash codes and Enum's:
https://stackoverflow.com/questions/4885095/what-is-the-reason-behind-enum-hashcode

And some issues (most of them rejected) at the Oracle Bug Java database:

• http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8050217
• http://bugs.java.com/bugdatabase/view_bug.do?bug_id=7190798

Issue Links

• is related to: SPARK-597 HashPartitioner incorrectly partitions RDD[Array[_]]
• links to: [Github] Pull Request #3795 (JoshRosen)