[SPARK-24259] ArrayWriter for Arrow produces wrong output

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: PySpark, SQL
• Labels: correctness
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 13/May/18 00:28
• Updated: 15/May/18 15:49
• Resolved: 15/May/18 14:15

Description

Right now ArrayWriter used to output Arrow data for array type, doesn't do clear or reset after each batch. It produces wrong output.

Issue Links

• links to: [Github] Pull Request #21312 (viirya)