XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The current byte comparison implementation in Hive is basic. It handles a byte (1 byte) at a time, so it's slow.

      There's FastByteComparisons class in Hadoop with sun.misc.Unsafe class. (https://github.com/hanborq/hadoop/blob/master/src/core/org/apache/hadoop/io/FastByteComparisons.java) It handles a long integer (8 bytes) at a time, so it's faster.

      Java 8 has String.compare, String.equalTo intrinsics with AVX2 and SSE4.2. (http://hg.openjdk.java.net/jdk8/jdk8/hotspot/rev/038dd2875b94) It handles 128~256 bits (16~32 bytes) at a time, so it's much faster.

      However, Unsafe.getLong and String.compare intrinsic needs additional data copies, so the actual performance increase is smaller than "1 byte : 32 bytes" comparison.

        Attachments

        1. HIVE-12348.patch
          19 kB
          Teddy Choi

          Issue Links

            Activity

              People

              • Assignee:
                teddy.choi Teddy Choi
                Reporter:
                teddy.choi Teddy Choi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: