[SPARK-16620] RDD.pipe(command: String) in Spark 2.0 does not work when command is specified with some options

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Liwei Lin(Inactive)
• Reporter: Liwei Lin(Inactive)
• Created: 19/Jul/16 06:06
• Updated: 19/Jul/16 17:25
• Resolved: 19/Jul/16 17:25

Description

Currently RDD.pipe(command: String):

• works only when the command is specified without any options, such as RDD.pipe("wc")
• does NOT work when the command is specified with some options, such as RDD.pipe("wc -l")

This is a regression from Spark 1.6.

Issue Links

• is related to: SPARK-16613 RDD.pipe returns values for empty partitions
• links to: [Github] Pull Request #14256 (lw-lin)