[ARROW-15820] [C++][Doc] Add table_source to streaming_execution.rst & clarify parameter name

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/31261
• Assignee: Vibhatha Lakmal Abeykoon
• Reporter: Weston Pace
• Created: 02/Mar/22 03:48
• Updated: 11/Jan/23 11:39
• Resolved: 19/Mar/22 01:55

Description

Currently the table_source node does not appear in our documentation.

Also, in TableSourceNodeOptions we have:

  // Size of batches to emit from this node
  // If the table is larger the node will emit multiple batches from the
  // the table to be processed in parallel.
  int64_t batch_size;

However, when looking into a performance issue today, I realized this description is incomplete. In reality we should probably call this parameter max_batch_size.

Furthermore, we should make it clear that a table with smaller batches will emit smaller batches directly (this is a good thing in my case) and will not concatenate small batches together into a larger batch.

Issue Links

• links to: GitHub Pull Request #12555