[SPARK-9460] Avoid byte array allocation in StringPrefixComparator

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 30/Jul/15 00:52
• Updated: 30/Jul/15 19:02
• Resolved: 30/Jul/15 04:19

Description

StringPrefixComparator converts the long values back to byte arrays in order to compare them. We should be able to optimize this to compare the longs directly, rather than turning the longs into byte arrays and comparing them byte by byte.

    public int compare(long aPrefix, long bPrefix) {
      // TODO: can done more efficiently
      byte[] a = Longs.toByteArray(aPrefix);
      byte[] b = Longs.toByteArray(bPrefix);
      for (int i = 0; i < 8; i++) {
        int c = UnsignedBytes.compare(a[i], b[i]);
        if (c != 0) return c;
      }
      return 0;
    }

Issue Links

• is related to: SPARK-9483 UTF8String.getPrefix only works in little-endian order
• links to: [Github] Pull Request #7765 (rxin)
• links to: [Github] Pull Request #7789 (rxin)