Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5166 Stabilize Spark SQL APIs
  3. SPARK-5193

Make Spark SQL API usable in Java and remove the Java-specific API

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Java version of the SchemaRDD API causes high maintenance burden for Spark SQL itself and downstream libraries (e.g. MLlib pipeline API needs to support both JavaSchemaRDD and SchemaRDD). We can audit the Scala API and make it usable for Java, and then we can remove the Java specific version.

      Things to remove include (Java version of):

      • data type
      • Row
      • SQLContext
      • HiveContext

      Things to consider:

      • Scala and Java have a different collection library.
      • Scala and Java (8) have different closure interface.
      • Scala and Java can have duplicate definitions of common classes, such as BigDecimal.

        Attachments

          Activity

            People

            • Assignee:
              rxin Reynold Xin
              Reporter:
              rxin Reynold Xin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: