[IMPALA-1699] HdfsPartition's compareTo() method breaks contract

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.1
• Fix Version/s: Impala 2.2
• Labels: compute-stats ramp-up
• Assignee: Juan Yu
• Reporter: Henry Robinson
• Created: 26/Jan/15 23:11
• Updated: 04/Jan/17 23:58
• Resolved: 03/Mar/15 18:49

Description

During SHOW COLUMN STATS you might occasionally see:

'ERROR: IllegalArgumentException: Comparison method violates its general contract!'

The stack trace is:

at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:835)
	at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:453)
	at java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:392)
	at java.util.ComparableTimSort.sort(ComparableTimSort.java:191)
	at java.util.ComparableTimSort.sort(ComparableTimSort.java:146)
	at java.util.Arrays.sort(Arrays.java:472)
	at java.util.Collections.sort(Collections.java:155)
	at com.cloudera.impala.catalog.HdfsTable.getTableStats(HdfsTable.java:1342)
	at com.cloudera.impala.service.Frontend.getTableStats(Frontend.java:631)
	at com.cloudera.impala.service.JniFrontend.getStats(JniFrontend.java:293)

The culprit is HdfsPartition's compareTo() method.

@Override
  public int compareTo(HdfsPartition o) {
    int sizeDiff = partitionKeyValues_.size() - o.getPartitionValues().size();
    if (sizeDiff != 0) return sizeDiff;
    for (int i = 0; i < partitionKeyValues_.size(); ++i) {
      int cmp = partitionKeyValues_.get(i).compareTo(o.getPartitionValues().get(i));
      if (cmp != 0) return cmp;
    }
    return 0;
  }