[ARROW-15282] [C++][FlightRPC] Support non-grpc data planes

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++, FlightRPC
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30773
• Assignee: David Li
• Reporter: Yibo Cai
• Created: 07/Jan/22 13:58
• Updated: 11/Jan/23 11:36
• Resolved: 11/Mar/22 02:54

Description

The goal is to decouple FlightData payload transmission from gRPC and provide a convenient framework to support other accelerated data planes (ucx, dpdk, xdp, etc.).

As a PoC, I implemented a shared memory data plane for local rpc, the performance is great compared with current best (grpc over unix socket).

Tested throughput and latency on xeon gold 5218, DoGet, batch size 128K.

streams	grpc over unix socket	shared memory data plane
1	3324 MB/s, 35 us	7045 MB/s, 16 us
2	6289 MB/s, 38 us	13311 MB/s, 17 us
4	10037 MB/s, 44 us	25012 MB/s, 17 us

Issue Links

• is duplicated by: ARROW-13889 [C++][FlightRPC] Decouple Arrow Flight RPC from GRPC
• is related to: ARROW-15835 [C++][FlightRPC] Refactor auth, middleware into the transport-agnostic layer
• is related to: ARROW-15836 [C++][FlightRPC] Refactor remaining methods into transport-agnostic handlers
• is related to: ARROW-15788 [C++][FlightRPC] Support alternative transports in the Flight benchmark
• is superceded by: ARROW-15706 [C++][FlightRPC] Implement a UCX transport
• links to: GitHub Pull Request #12196
• links to: GitHub Pull Request #12465