Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5166

Stabilize Spark SQL APIs

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: SQL
    • Labels:
    • Target Version/s:

      Description

      Before we take Spark SQL out of alpha, we need to audit the APIs and stabilize them.

      As a general rule, everything under org.apache.spark.sql.catalyst should not be exposed.

        Attachments

        Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-6116 DataFrame API improvement umbrella ticket (Spark 1.5)

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. SPARK-4988 "Create table ..as select ..from..order by .. limit 10" report error when one col is a Decimal

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SPARK-2096 Correctly parse dot notations for accessing an array of structs

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-5180 Data source API improvement (Spark 1.5)

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. SPARK-5135 Add support for describe [extended] table to DDL in SQLContext

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. SPARK-5061 SQLContext: overload createParquetFile

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-4945 Add overwrite option support for SchemaRDD.saveAsParquetFile

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        links to

        Pull request #4642 [Github] Pull Request #4642 (marmbrus)

        1.
        		 Stabilize Spark SQL data type API Sub-task Resolved Reynold Xin Actions
        2.
        		 Move Row into sql package and make it usable for Java Sub-task Resolved Reynold Xin Actions
        3.
        		 Adding data frame APIs to SchemaRDD Sub-task Resolved Reynold Xin Actions
        4.
        		 Make SQLConf a field rather than mixin in SQLContext Sub-task Resolved Reynold Xin Actions
        5.
        		 Native Date type for SQL92 Date Sub-task Resolved Adrian Wang Actions
        6.
        		 [SQL] Public API in SQLContext to list tables Sub-task Resolved Bill Bejeck Actions
        7.
        		 Make Spark SQL API usable in Java and remove the Java-specific API Sub-task Resolved Reynold Xin Actions
        8.
        		 Move Decimal from types.decimal to types package Sub-task Resolved Adrian Wang Actions
        9.
        		 Enable javadoc/scaladoc for public classes in catalyst project Sub-task Resolved Michael Armbrust Actions
        10.
        		 Determine serializability of SQLContext Sub-task Resolved Reynold Xin Actions
        11.
        		 Clean up exposed classes in sql.hive package Sub-task Resolved Reynold Xin Actions
        12.
        		 Stabilize UDFRegistration API Sub-task Resolved Reynold Xin Actions
        13.
        		 Use java.math.BigDecimal as the exposed Decimal type Sub-task Resolved Reynold Xin Actions
        14.
        		 Update SQL programming guide for 1.3 Sub-task Resolved Unassigned Actions
        15.
        		 Row shouldn't extend Seq Sub-task Resolved Reynold Xin Actions
        16.
        		 Add defaultSizeOf to every data type Sub-task Resolved Yin Huai Actions
        17.
        		 Cross-langauge load/store functions for creating and saving DataFrames Sub-task Resolved Yin Huai Actions
        18.
        		 Make sure DataFrame expressions are usable in Java Sub-task Resolved Reynold Xin Actions
        19.
        		 Replace reference to SchemaRDD with DataFrame Sub-task Resolved Reynold Xin Actions
        20.
        		 Make CacheManager a concrete class and field in SQLContext Sub-task Resolved Reynold Xin Actions
        21.
        		 Remove Python LocalHiveContext Sub-task Resolved Reynold Xin Actions
        22.
        		 Break sql.py into multiple files Sub-task Resolved Davies Liu Actions
        23.
        		 SQLContext.createDataFrame shouldn't be an implicit function Sub-task Closed Reynold Xin Actions
        24.
        		 collect should call executeCollect Sub-task Resolved Reynold Xin Actions
        25.
        		 Error messages for plans with invalid AttributeReferences Sub-task Resolved Michael Armbrust Actions
        26.
        		 Create type alias for SchemaRDD for source backward compatibility Sub-task Resolved Reynold Xin Actions
        27.
        		 Support explode in DataFrame DSL Sub-task Resolved Michael Armbrust Actions
        28.
        		 Add more tests and docs for DataFrame Python API Sub-task Resolved Davies Liu Actions
        29.
        		 Create a convenient way for Python users to register SQL UDFs Sub-task Resolved Davies Liu Actions
        30.
        		 Provide a convenient way for Scala users to use UDFs Sub-task Resolved Reynold Xin Actions
        31.
        		 Provide support for project using SQL expression Sub-task Resolved Reynold Xin Actions
        32.
        		 support select/filter by SQL expression for Python DataFrame Sub-task Resolved Davies Liu Actions
        33.
        		 Better support for creating DataFrame from local data collection Sub-task Resolved Reynold Xin Actions
        34.
        		 Allow using String to specify colum name in DSL aggregate functions Sub-task Resolved Reynold Xin Actions
        35.
        		 Move DataFrame implicit functions into SQLContext.implicits Sub-task Resolved Reynold Xin Actions
        36.
        		 Add a config flag to disable eager analysis of DataFrames Sub-task Resolved Reynold Xin Actions
        37.
        		 Support DataFrame.renameColumn Sub-task Resolved Reynold Xin Actions
        38.
        		 Add a show method to print the content of a DataFrame in columnar format Sub-task Resolved Reynold Xin Actions
        39.
        		 XyzType companion object should subclass XyzType Sub-task Resolved Reynold Xin Actions
        40.
        		 Python DataFrame API remaining tasks Sub-task Resolved Davies Liu Actions
        41.
        		 DataFrame.to_pandas Sub-task Resolved Davies Liu Actions
        42.
        		 Allow short names for built-in data sources Sub-task Resolved Reynold Xin Actions
        43.
        		 createDataFrame replace applySchema/inferSchema Sub-task Resolved Davies Liu Actions
        44.
        		 Allow creating a DataFrame from local Python data Sub-task Resolved Davies Liu Actions
        45.
        		 Don't implicitly convert RDDs directly to DataFrames Sub-task Resolved Reynold Xin Actions
        46.
        		 Schema support in Row Sub-task Resolved Reynold Xin Actions
        47.
        		 fix Data Frame Python API Sub-task Resolved Davies Liu Actions
        48.
        		 DataFrame methods with varargs do not work in Java Sub-task Resolved Reynold Xin Actions
        49.
        		 sortBy -> orderBy in Python Sub-task Resolved Reynold Xin Actions
        50.
        		 Python DataFrame documentation fixes Sub-task Resolved Davies Liu Actions

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment