Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25737

Remove JavaSparkContextVarargsWorkaround and standardize union() methods

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
    • Target Version/s:
    • Docs Text:
      Hide
      For release notes:

      Before Spark 3.0, JavaSparkContext exposed union() methods with signatures like union(JavaRDD, List<JavaRDD>). This was a workaround for a compiler issue that is since resolved. The signature of union() methods in JavaSparkContext has changed to simple varargs: union(JavaRDD*).
      Show
      For release notes: Before Spark 3.0, JavaSparkContext exposed union() methods with signatures like union(JavaRDD, List<JavaRDD>). This was a workaround for a compiler issue that is since resolved. The signature of union() methods in JavaSparkContext has changed to simple varargs: union(JavaRDD*).

      Description

      In ancient times in 2013, JavaSparkContext got a superclass JavaSparkContextVarargsWorkaround to deal with some Scala 2.7 issue: http://www.scala-archive.org/Workaround-for-implementing-java-varargs-in-2-7-2-final-td1944767.html#a1944772

      I believe this was really resolved by the @varags annotation in Scala 2.9. 

      I believe we can now remove this workaround. union(JavaRDD, List<JavaRDD>) should just be union(JavaRDD*) and likewise for JavaPairRDD, JavaDoubleRDD.

       

        Attachments

          Activity

            People

            • Assignee:
              srowen Sean R. Owen
              Reporter:
              srowen Sean R. Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: