[SAMZA-1516] Another round of issues found by BEAM tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Assignee: Xinyu Liu
• Reporter: Xinyu Liu
• Created: 28/Nov/17 21:42
• Updated: 03/Dec/17 16:31
• Resolved: 03/Dec/17 16:31

Description

A couple of more fixes:

1. fix a bug of identifying input streams for an operator. The logic is inside the traversal of operator graph inside OperatorImplGraph. Basically we need to traverse further to add the input stream to the operator even the operator is already created.

2. for partitionBy, right now if there is null key and null partition key, Kafka will do round-robin (https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html). This is causing problems for BEAM reduce. Since BEAM assume records without key will go to a single reducer. Add a fix in partitonBy to set the partitionKey.

Issue Links

• links to: GitHub Pull Request #370