[FLINK-3290] [py] Generalize OperationInfo transfer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1
• Fix Version/s: 1.0.0
• Component/s: API / Python
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 26/Jan/16 11:41
• Updated: 12/Feb/16 14:05
• Resolved: 27/Jan/16 14:08

Description

A set number of arguments is transferred whenever a user defines an operation. For a CSV Source for example these are delimiters/filepath, for a map function only the set ID'S are transferred. As such, for all operators a separate routine is defined that governs which arguments are transferred.

While working on FLINK-3275 I realized that adding a new argument/parameter, in this case parallelism, is not as straightforward as it could be. Most newly added operators will require a new routine; whereas adding new arguments may require the modification of multiple routines. Over times, this is bound to become a big mess.

All arguments are stored in an OperationInfo object, which also contains default values for all unused arguments. I want to generalize the whole affair by transferring all arguments, used or not.

This will reduce clutter, make it easier to add new parameters (only 4 new lines needed, 2 for defining new fields inside Java/Python OperationInfo Classes; 1 each for sending/receiving the new argument) and will make the transfer consistent across all operations.