Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: io, performance, util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Guava has a nice implementation of lexicographical byte-array comparison that uses sun.misc.Unsafe to compare unsigned byte arrays long-at-a-time. Their benchmarks show it as being 2x more CPU-efficient than the equivalent pure-Java implementation. We can easily integrate this into WritableComparator.compareBytes to improve CPU performance in the shuffle.

      1. hadoop-7761.txt
        8 kB
        Todd Lipcon
      2. hadoop-7761.txt
        9 kB
        Todd Lipcon
      3. hadoop-7761.txt
        10 kB
        Todd Lipcon
      4. hadoop-7761.txt
        10 kB
        Todd Lipcon

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development