[ARROW-300] [Format] Add body buffer compression option to IPC message protocol using LZ4 or ZSTD

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Format
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/15325
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 21/Sep/16 20:29
• Updated: 11/Jan/23 07:07
• Resolved: 01/May/20 00:41

Description

It may be useful if data is to be sent over the wire to compress the data buffers themselves as their being written in the file layout.

I would propose that we keep this extremely simple with a global buffer compression setting in the file Footer. Probably only two compressors worth supporting out of the box would be zlib (higher compression ratios) and lz4 (better performance).

What does everyone think?

Issue Links

• is related to: ARROW-1099 [C++] Add support for PFOR integer compression
• is related to: ARROW-5821 [Java] Support compact fixed-width vectors
• links to: GitHub Pull Request #70
• links to: GitHub Pull Request #6707

Sub-Tasks

• [C++] Implement experimental buffer compression in IPC messages: Resolved