[MRQL-101] Fix memory materialization of lazy bags

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.8
• Component/s: Run-Time/Flink, Run-Time/Spark
• Assignee: Leonidas Fegaras
• Reporter: Leonidas Fegaras
• Created: 26/Jan/17 19:33
• Updated: 28/Jan/17 14:02
• Resolved: 28/Jan/17 14:02

Description

This patch addresses various bugs in bag Materialization. Materializing lazy bags (stream-based bags implemented as iterators) is needed in a MRQL plan If the plan returns a lazy Bag and the term occurs in the code multiple times. In that case, we generate code to materialize this Bag in memory (as an array). The new materialization algorithm in this patch is simpler and more general, and addresses various bugs which result to the error "The collection stream has already been consumed".

Issue Links

• links to: GitHub Pull Request #33