Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-9600

[Rust] When used as a crate dependency, arrow-flight is rebuilt on every invocation of cargo build

    XMLWordPrintableJSON

    Details

      Description

      When used as a crate dependency, arrow-flight is rebuilt on every invocation of cargo build

      Repro:

      Create a new repo, add `arrow=1.0.0` as a dependency, and then run `cargo build`

      Expected behavior: After the first successful invocation of `cargo build`, arrow-flight will not recompile if no other changes are made.

      Actual behavior: After every invocation of `cargo build`, arrow-flight is recompiled, even when nothing has changed

      Example

       
      Create a new crate

       alamb@ip-192-168-0-129 arrow_rebuilds % cargo new too_many_rebuilds --bin
       cargo new too_many_rebuilds --bin
       Created binary (application) `too_many_rebuilds` package
      

      Add arrow as a dependency in Cargo.toml:

       diff --git a/Cargo.toml b/Cargo.toml
       index a239680..44ed358 100644
       — a/Cargo.toml
       +++ b/Cargo.toml
       @@ -5,3 +5,6 @@ authors = ["alamb <andrew@nerdnetworks.org>"]
       edition = "2018"
       # See more keys and their definitions at [https://doc.rust-lang.org/cargo/reference/manifest.html]
       +
       +[dependencies]
       +arrow = "1.0.0"
      

      Now, all invocations of `cargo build` will rebuild arrow, even though nothing in the code has changed:

       alamb@ip-192-168-0-129 too_many_rebuilds % cargo build
       cargo build
       Compiling arrow-flight v1.0.0
       Compiling arrow v1.0.0
       Compiling too_many_rebuilds v0.1.0 (/Users/alamb/Software/bugs/arrow_rebuilds/too_many_rebuilds)
       Finished dev [unoptimized + debuginfo] target(s) in 8.70s
       alamb@ip-192-168-0-129 too_many_rebuilds % cargo build
       cargo build
       Compiling arrow-flight v1.0.0
       Compiling arrow v1.0.0
       Compiling too_many_rebuilds v0.1.0 (/Users/alamb/Software/bugs/arrow_rebuilds/too_many_rebuilds)
       Finished dev [unoptimized + debuginfo] target(s) in 8.65s
      

      You can see what is happening by checking out a fresh copy of arrow/master (no Cargo.log) and running `cargo build` – you'll see your local checkout has changes in rust/arrow-flight/src/arrow.flight.protocol.rs:

       alamb@ip-192-168-0-129 arrow % cd rust/arrow
       cd rust/arrow
       alamb@ip-192-168-0-129 arrow % git status
       git status
       On branch master
       Your branch is up to date with 'origin/master'.
      
      nothing to commit, working tree clean
       alamb@ip-192-168-0-129 arrow % cargo build
       cargo build
       Compiling futures-task v0.3.5
       ...
       Compiling arrow v2.0.0-SNAPSHOT (/Users/alamb/Software/arrow/rust/arrow)
       Finished dev [unoptimized + debuginfo] target(s) in 21.76s
       alamb@ip-192-168-0-129 arrow %
      
      alamb@ip-192-168-0-129 arrow % git status
       git status
       On branch master
       Your branch is up to date with 'origin/master'.
      
      Changes not staged for commit:
       (use "git add <file>..." to update what will be committed)
       (use "git restore <file>..." to discard changes in working directory)
       modified: ../arrow-flight/src/arrow.flight.protocol.rs
      
      no changes added to commit (use "git add" and/or "git commit -a")
      
      1. Root Cause Analysis

      The issue is that the build.rs of arrow-flight calls `tonic_build` to auto generate `rust/arrow-flight/src/arrow.flight.protocol.rs`, which is also checked in (first done in https://github.com/apache/arrow/commit/ec84b7b8102f227295f865c420496830c66a6281).

      This file and the version of tonic were updated on https://github.com/apache/arrow/commit/7b49cbc23f22ed99eebf85cc0b9acb1f0d3f832f on July 11, 2020

      It turns out that the output of "tonic_build" depends on not only on the version of tonic, but also on the version of proc-macro2, and the version of proc-macro2 is not specifically pinned.

      `proc-macro2 = "1.0.19"` was released on July 19, 2020 (https://crates.io/crates/proc-macro2/1.0.19) and it appears to subtlety changes the resulting output of arrow.flight.protocol.rs; Thus the output no longer matches what is checked in. This means that anyone without a Cargo.lock file that pins proc-macro2 to 1.0.18 would get 1.0.19 and thus also a local modification during build.

      Workaround

      If we pin Cargo.toml to use proc-macro2 1.0.18 the local modification stops.

       proc-macro2 = "1.0.18"
       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                andygrove Andy Grove
                Reporter:
                alamb Andrew Lamb
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h