[MADLIB-1224] Select default buffer size for mini-batch preprocessor

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.14
• Component/s: Module: Utilities
• Assignee: Jingyi Mei
• Reporter: Jingyi Mei
• Created: 05/Apr/18 00:08
• Updated: 13/Apr/18 16:32
• Resolved: 13/Apr/18 16:32

Description

As a follow up of https://issues.apache.org/jira/browse/MADLIB-1200

 

In minibatch_preprocessor, we made buffer_size as an optional parameter. If it is not set, some default value will be assigned. Current considerations are:

1. Within segment, each cell has 1GB limit so that we can't put too many rows into one super row to exceed the limit
2. Among segments, data should be distributed as equally as possible to avoid data skew so that GPDB can work more efficiently.