[SPARK-2991] RDD transforms for scan and scanLeft

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: Spark Core
• Labels: features
• Assignee: Erik Erlandson
• Reporter: Erik Erlandson
• Created: 12/Aug/14 19:08
• Updated: 03/Nov/15 15:31
• Resolved: 16/Sep/15 18:40

Description

Provide RDD transforms analogous to Scala scan(z)(f) (parallel prefix scan) and scanLeft(z)(f) (sequential prefix scan)

Discussion of a scanLeft implementation:
http://erikerlandson.github.io/blog/2014/08/09/implementing-an-rdd-scanleft-transform-with-cascade-rdds/

Discussion of scan:
http://erikerlandson.github.io/blog/2014/08/12/implementing-parallel-prefix-scan-as-a-spark-rdd-transform/

Issue Links

• relates to: SPARK-9999 Dataset API on top of Catalyst/DataFrame
• links to: [Github] Pull Request #1909 (erikerlandson)