[ARROW-9600] [Rust] When used as a crate dependency, arrow-flight is rebuilt on every invocation of cargo build

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 2.0.0
• Component/s: Rust
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25663
• Assignee: Andy Grove
• Reporter: Andrew Lamb
• Created: 30/Jul/20 14:21
• Updated: 11/Jan/23 08:07
• Resolved: 03/Aug/20 16:06

Description

When used as a crate dependency, arrow-flight is rebuilt on every invocation of cargo build

Repro:

Create a new repo, add `arrow=1.0.0` as a dependency, and then run `cargo build`

Expected behavior: After the first successful invocation of `cargo build`, arrow-flight will not recompile if no other changes are made.

Actual behavior: After every invocation of `cargo build`, arrow-flight is recompiled, even when nothing has changed

Example

 
Create a new crate

 alamb@ip-192-168-0-129 arrow_rebuilds % cargo new too_many_rebuilds --bin
 cargo new too_many_rebuilds --bin
 Created binary (application) `too_many_rebuilds` package

Add arrow as a dependency in Cargo.toml:

 diff --git a/Cargo.toml b/Cargo.toml
 index a239680..44ed358 100644
 — a/Cargo.toml
 +++ b/Cargo.toml
 @@ -5,3 +5,6 @@ authors = ["alamb <[e-mail]>"]
 edition = "2018"
 # See more keys and their definitions at [https://doc.rust-lang.org/cargo/reference/manifest.html]
 +
 +[dependencies]
 +arrow = "1.0.0"

Now, all invocations of `cargo build` will rebuild arrow, even though nothing in the code has changed:

 alamb@ip-192-168-0-129 too_many_rebuilds % cargo build
 cargo build
 Compiling arrow-flight v1.0.0
 Compiling arrow v1.0.0
 Compiling too_many_rebuilds v0.1.0 (/Users/alamb/Software/bugs/arrow_rebuilds/too_many_rebuilds)
 Finished dev [unoptimized + debuginfo] target(s) in 8.70s
 alamb@ip-192-168-0-129 too_many_rebuilds % cargo build
 cargo build
 Compiling arrow-flight v1.0.0
 Compiling arrow v1.0.0
 Compiling too_many_rebuilds v0.1.0 (/Users/alamb/Software/bugs/arrow_rebuilds/too_many_rebuilds)
 Finished dev [unoptimized + debuginfo] target(s) in 8.65s

You can see what is happening by checking out a fresh copy of arrow/master (no Cargo.log) and running `cargo build` – you'll see your local checkout has changes in rust/arrow-flight/src/arrow.flight.protocol.rs:

 alamb@ip-192-168-0-129 arrow % cd rust/arrow
 cd rust/arrow
 alamb@ip-192-168-0-129 arrow % git status
 git status
 On branch master
 Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
 alamb@ip-192-168-0-129 arrow % cargo build
 cargo build
 Compiling futures-task v0.3.5
 ...
 Compiling arrow v2.0.0-SNAPSHOT (/Users/alamb/Software/arrow/rust/arrow)
 Finished dev [unoptimized + debuginfo] target(s) in 21.76s
 alamb@ip-192-168-0-129 arrow %

alamb@ip-192-168-0-129 arrow % git status
 git status
 On branch master
 Your branch is up to date with 'origin/master'.

Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git restore <file>..." to discard changes in working directory)
 modified: ../arrow-flight/src/arrow.flight.protocol.rs

no changes added to commit (use "git add" and/or "git commit -a")

1. Root Cause Analysis

The issue is that the build.rs of arrow-flight calls `tonic_build` to auto generate `rust/arrow-flight/src/arrow.flight.protocol.rs`, which is also checked in (first done in https://github.com/apache/arrow/commit/ec84b7b8102f227295f865c420496830c66a6281).

This file and the version of tonic were updated on https://github.com/apache/arrow/commit/7b49cbc23f22ed99eebf85cc0b9acb1f0d3f832f on July 11, 2020

It turns out that the output of "tonic_build" depends on not only on the version of tonic, but also on the version of proc-macro2, and the version of proc-macro2 is not specifically pinned.

`proc-macro2 = "1.0.19"` was released on July 19, 2020 (https://crates.io/crates/proc-macro2/1.0.19) and it appears to subtlety changes the resulting output of arrow.flight.protocol.rs; Thus the output no longer matches what is checked in. This means that anyone without a Cargo.lock file that pins proc-macro2 to 1.0.18 would get 1.0.19 and thus also a local modification during build.

Workaround

If we pin Cargo.toml to use proc-macro2 1.0.18 the local modification stops.

 proc-macro2 = "1.0.18"
 

 

Issue Links

• is related to: ARROW-9628 [Rust] Clippy PR test failing intermittently on Rust / AMD64 MacOS
• links to: GitHub Pull Request #7867
• links to: GitHub Pull Request #7893