Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1448

Improve bitshuffle to enable AVX2 at runtime

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.8.0
    • 1.2.0
    • cfile, perf
    • None

    Description

      I've been doing a little testing with the BITSHUFFLE encoding, and found that scan performance is improved about 30% on a Haswell cluster by enabling AVX2 when building the bitshuffle library. Unfortunately, that would cause a crash on earlier boxes, so we can't ship like that.

      I filed https://github.com/kiyo-masui/bitshuffle/issues/41 to do the switching at runtime.

      Attachments

        1. Selection_029.png
          59 kB
          Todd Lipcon

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. KUDU-1751 Use better encodings by default

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: