[SPARK-2315] drop, dropRight and dropWhile which take RDD input and return RDD

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark Core
• Labels: features
• Assignee: Erik Erlandson
• Reporter: Erik Erlandson
• Created: 28/Jun/14 04:45
• Updated: 09/Aug/16 23:14
• Resolved: 14/Jun/15 01:16

Description

Last time I loaded in a text file, I found myself wanting to just skip the first element as it was a header. I wrote candidate methods drop, dropRight and dropWhile to satisfy this kind of need:

val txt = sc.textFile("text_with_header.txt")
val data = txt.drop(1)

Issue Links

• is related to: SPARK-3998 Add offset limit to SqlParser.scala
• relates to: SPARK-9999 Dataset API on top of Catalyst/DataFrame
• links to: [Github] Pull Request #1254 (erikerlandson)
• links to: [Github] Pull Request #1839 (erikerlandson)