Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10453

[Rust] [DataFusion] Performance degredation after removing specialization



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Rust, Rust - DataFusion
    • None


      The following commit caused a pretty large drop in performance for the TPC-H benchmark running against a SF=100 data set.

       29e9d13481ea6acc3f74cda108ed34ef8a411ba2 is the first bad commit
      commit 29e9d13481ea6acc3f74cda108ed34ef8a411ba2
      Author: Jorge C. Leitao <jorgecarleitao@gmail.com>
      Date:   Sun Oct 18 21:05:48 2020 +0200    ARROW-10002: [Rust] Remove trait specialization from arrow crate
          This PR removes trait specialization by leveraging the compiler to remove trivial `if` statements.
          I verified that the assembly code was the same in a [simple example](https://rust.godbolt.org/z/qrcW8W). I do not know if this generalizes to our use-case, but I suspect so as LLVM is (hopefully) removing trivial branches like `if a != a`.
          The change `get_data_type()` to `DATA_TYPE` is not necessary. I did it before realizing this. IMO it makes it more explicit that this is not a function, but a constant, but we can revert it.
          Closes #8485 from jorgecarleitao/simp_types
          Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
          Signed-off-by: Neville Dipale <nevilledips@gmail.com>:040000 040000 cbdaf3c9e924ec0e51d178df73169956b2bf723f 87c79e17378196b61dce9c5373e008ee94620d58 M	rust

      Benchmark command:

       cargo run --release --bin tpch -- --iterations 3 --path /mnt/tpch/parquet-100GB --format parquet --query 1 --batch-size 4096 --concurrency 24

      Before this commit:

      Query 1 iteration 0 took 13629 ms
      Query 1 iteration 1 took 13450 ms
      Query 1 iteration 2 took 13465 ms 

      After this commit:

      Query 1 iteration 0 took 18586 ms
      Query 1 iteration 1 took 18297 ms
      Query 1 iteration 2 took 18253 ms 


        Issue Links



              jorgecarleitao Jorge Leitão
              andygrove Andy Grove
              0 Vote for this issue
              4 Start watching this issue