Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9486

Add aliasing to data sources to allow external packages to register themselves with Spark

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:
    • Sprint:
      Spark 1.5 release

      Description

      Currently Spark allows users to use external data sources like spark-avro, spark-csv, etc by having them specifying their full class name:

      sqlContext.read.format("com.databricks.spark.avro").load(path)
      

      Typing in a full class is not the best idea so it would be nice to allow the external packages to be able to register themselves with Spark to allow users to do something like:

      sqlContext.read.format("avro").load(path)
      

      This would make it so that the external data source packages follow the same convention as the built in data sources do, parquet, json, jdbc, etc.

      This could be accomplished by using a ServiceLoader.

        Attachments

          Activity

            People

            • Assignee:
              jd Joseph Batchik
              Reporter:
              jd Joseph Batchik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: