[PIG-4012] java.lang.IllegalArgumentException: Comparison method violates its general contract! SpillableMemoryManager

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.15.0
• Component/s: impl

Environment

java version "1.7.0_60-ea"
Java(TM) SE Runtime Environment (build 1.7.0_60-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b04, mixed mode)

• Assignee: Rohini Palaniswamy
• Reporter: David Dreyfus
• Created: 13/Jun/14 13:53
• Updated: 23/May/15 18:33
• Resolved: 16/Jun/14 12:50

Description

java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
at java.util.TimSort.sort(TimSort.java:223)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:199)
at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
at sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:301)
at sun.management.Sensor.trigger(Sensor.java:137)

From SpillableMemoryManager.java:
/**

• We don't lock anything, so this sort may not be stable if a WeakReference suddenly
• becomes null, but it will be close enough.
• Also between the time we sort and we use these spillables, they
• may actually change in size - so this is just best effort
  */
  Issue may be due to Java 7 and reporting vs ignoring the exception.
  Trying
  -Djava.util.Arrays.useLegacyMergeSort=true

http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6804124
suggests the newer MergeSort is much faster.

Someone may want to make the sorting stable in SpillableMemoryManager so that the new merge sort can be used without failure.

Issue Links

• is part of: PIG-4564 Pig can deadlock in POPartialAgg if there is a bag