[SPARK-21967] org.apache.spark.unsafe.types.UTF8String#compareTo Should Compare 8 Bytes at a Time for Better Performance

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Labels: perfomance
• Assignee: Armin Braun
• Reporter: Armin Braun
• Created: 10/Sep/17 13:17
• Updated: 16/Sep/17 08:18
• Resolved: 16/Sep/17 08:18

Description

org.apache.spark.unsafe.types.UTF8String#compareTo contains the following TODO:

    int len = Math.min(numBytes, other.numBytes);
    // TODO: compare 8 bytes as unsigned long
    for (int i = 0; i < len; i ++) {
      // In UTF-8, the byte should be unsigned, so we should compare them as unsigned int.

The todo should be resolved by comparing the maximum number of 64bit words possible in this method, before falling back to unsigned int comparison.

Issue Links

• links to: [Github] Pull Request #19180 (original-brownbear)