[ARROW-6625] [Python] Allow concat_tables to null or default fill missing columns

• Type: Wish
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22978
• Assignee: Zhuo Peng
• Reporter: Daniel Nugent
• Created: 19/Sep/19 21:02
• Updated: 11/Jan/23 07:48
• Resolved: 08/Nov/19 17:33

Description

The concat_tables function currently requires schemas to be identical across all tables to be concat'ed together. However, tables occasionally are conforming on type where present, but a column will be absent.

In this case, allowing for null filling (or default filling) would be ideal.

I imagine this feature would be an optional parameter on the concat_tables function. Presumably the argument could be either a boolean in the case of blanket null filling, or a mapping type for default filling. If a user wanted to default fill some columns, but null fill others, they could use a None as the value (defaultdict would make it simple to provide a blanket null fill if only a few default value columns were desired).

If a mapping wasn't present, the function should probably raise an error.

The default behavior would be the current and thus the default value of the parameter should be False or None.

Issue Links

• supercedes: ARROW-7229 [C++] Unify ConcatenateTables APIs
• links to: GitHub Pull Request #5534
• links to: GitHub Pull Request #5804