Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12150

numPartitions argument to sqlContext.range() should be optional

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      It's a little inconsistent that the first two sqlContext.range() methods don't take a numPartitions arg, while the third one does.

      And more importantly, it's a little inconvenient that the numPartitions arg is mandatory for the third range() method - it means that if you want to specify a step, you suddenly have to think about partitioning - an orthogonal concern.

      My suggestion would be to make numPartitions optional, like it is on the sparkContext.range(..).

      Attachments

        Activity

          People

            smilegator Xiao Li
            henridf Henri DF
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: