[SPARK-7150] SQLContext.range()

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: ML, SQL
• Labels: starter
• Target Version/s: 1.4.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Adrian Wang
• Reporter: Joseph K. Bradley
• Created: 26/Apr/15 06:54
• Updated: 23/Mar/19 22:35
• Resolved: 19/May/15 04:43

Description

It would be handy to have easy ways to construct random columns for DataFrames. Proposed API:

class SQLContext {
  // Return a DataFrame with a single column named "id" that has consecutive value from 0 to n.
  def range(n: Long): DataFrame

  def range(n: Long, numPartitions: Int): DataFrame
}

Usage:

// uniform distribution
ctx.range(1000).select(rand())

// normal distribution
ctx.range(1000).select(randn())

We should add an RangeIterator that supports long start/stop position, and then use it to create an RDD as the basis for this DataFrame.

Issue Links

• is blocked by: SPARK-7248 Random number generators for DataFrames
• links to: [Github] Pull Request #6081 (adrian-wang)
• links to: [Github] Pull Request #6230 (davies)
• links to: [Github] Pull Request #6233 (adrian-wang)