[SPARK-6116] DataFrame API improvement umbrella ticket (Spark 1.5)

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Labels: DataFrame
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 02/Mar/15 21:08
• Updated: 11/Sep/15 17:20
• Resolved: 11/Sep/15 10:20

Description

An umbrella ticket for DataFrame API improvements for Spark 1.5.

SPARK-9576 is the ticket for Spark 1.6.

Issue Links

• relates to: SPARK-6956 Improve DataFrame API compatibility with Pandas
• relates to: SPARK-5166 Stabilize Spark SQL APIs
• relates to: SPARK-7239 Statistic functions for DataFrames
• relates to: SPARK-8002 Support virtual columns in SQL and DataFrames

Sub-Tasks

• describe function for summary statistics: Resolved
• DataFrame.dropna support: Resolved
• DataFrame.fillna: Resolved
• Add RDD methods to DataFrame to preserve schema: Resolved
• SQLContext.implicits should provide automatic conversion for RDD[Row]: Closed
• DataFrame.na.replace value support in Scala/Java: Resolved
• DataFrame.na.replace value support for Python: Resolved
• SQLContext.emptyDataFrame should contain 0 rows, not 1 row: Resolved
• SQLContext.registerFunction -> SQLContext.udf.register: Resolved
• Alias DataFrame.na.fill/drop in Python: Resolved
• Make DataFrame.rdd a lazy val: Resolved
• Decide on semantics for string identifiers in DataFrame API: Resolved
• not able to resolve dot('.') in field name: Resolved
• Join on two tables (generated from same one) is broken: Resolved
• Create a DataFrame join API to facilitate equijoin and self join: Resolved
• Missing alias function on Python DataFrame: Resolved
• Drop __getattr__ on pyspark.sql.DataFrame: Closed
• Stabilize Spark SQL data type API followup: Resolved
• Stabilize data types: Resolved
• UDF clean up: Resolved
• Remove PrimitiveType: Resolved
• Rename NativeType -> AtomicType: Resolved
• Clean up Python data type hierarchy: Resolved
• Implement struct, array, and map field accessor using apply in Scala and __getitem__ in Python: Resolved
• Expression for monotonically increasing IDs: Resolved
• Document what the options: Map[String, String] does on DataFrame.save and DataFrame.saveAsTable: Closed
• Support math functions in DataFrames: Resolved
• Support math functions in DataFrames in Python: Resolved
• SQLContext.range(): Resolved
• Correlation methods for DataFrame: Closed
• Add a Column expression for partition ID: Resolved
• Add randomSplit method to DataFrame: Resolved
• Add approximate stratified sampling to DataFrame: Resolved
• collect and take return different results: Resolved
• Make repartition and coalesce a part of the query plan: Resolved
• Support math functions in R DataFrame: Resolved
• Support fillna / dropna in R DataFrame: Resolved
• Create Column expression for array/struct creation: Resolved
• Add a method for dropping a column in Java/Scala: Resolved
• withColumn is very slow on dataframe with large number of columns: Resolved
• Audit missing Hive functions: Resolved
• Add Pandas' shift method to the Dataframe API: Closed
• Random number generators for DataFrames: Resolved
• Add a between function in Column: Resolved
• Add bitwise operations to DataFrame DSL: Resolved
• Improve the output from DataFrame.show(): Resolved
• Add rollup and cube support to DataFrame Java/Scala DSL: Resolved
• Add Column expression for conditional statements (if, case): Resolved
• Window function support in Scala/Java DataFrame DSL: Resolved
• Add DataFrame.dropDuplicates: Resolved
• Move mathfunctions into functions: Resolved
• Add coalesce Spark SQL function to PySpark API: Resolved
• By default retain group by columns in aggregate: Resolved
• Provide DataFrame.zip (analog of RDD.zip) to merge two data frames: Closed
• pyspark.sql.types.StructType and Row should implement __iter__(): Closed
• pyspark.sql.types.StructType.fromJson() is incorrectly named: Closed
• Add drop column to Python DataFrame API: Resolved
• Break dataframe.py into multiple files: Resolved
• Add explode expression: Resolved
• Don't split by dot if within backticks for DataFrame attribute resolution: Resolved
• Document all SQL/DataFrame public methods with @since tag: Resolved
• Document all PySpark SQL/DataFrame public methods with @since tag: Resolved
• DataFrameReader and DataFrameWriter for input/output API: Resolved
• make explode support struct type: Closed
• DataFrame reader/writer API in Python: Resolved
• Figure out what to do with insertInto w.r.t. DataFrameWriter API: Closed
• Add standard deviation aggregate expression: Closed
• Add rollup and cube support to DataFrame Python DSL: Resolved
• Window function support in Python DataFrame DSL: Resolved
• Better error for unresolved window functions.: Resolved
• DataFrame.ntile() should only accept Int as parameter: Resolved
• Add JavaDoc style deprecation for deprecated DataFrame methods: Resolved
• Support SQLContext.range(end): Resolved
• Improve DataFrame Python exception: Closed
• crosstab should use 0 instead of null for pairs that don't appear: Resolved
• Add methods to facilitate equi-join on multiple join keys: Resolved
• Python DataFrame: support passing a list into describe: Resolved
• Improve DataFrame.show() output: Resolved
• Improve frequent items documentation: Resolved
• DataFrameReader/Writer in Python does not match Scala: Resolved
• Add Column.alias to Scala/Java API: Resolved
• Design an easier way to construct schema for both Scala and Python: Resolved
• Improve Python reader/writer interface doc and testing: Resolved
• Better AnalysisException for writing DataFrame with identically named columns: Resolved
• DataFrame Python API: Alias replace in DataFrameNaFunctions: Resolved
• Improve error message reporting for DataFrame and SQL: Resolved
• DataFrame hint for broadcast join: Resolved
• Python DataFrameReader/Writer should mirror scala: Resolved
• Reconcile callUDF and callUdf: Resolved
• Prevent accidental use of "and" and "or" to build invalid expressions in Python: Resolved
• For PySpark's DataFrame API, we need to throw exceptions when users try to use and/or/not: Resolved
• expr function to convert SQL expression into a Column: Resolved
• dataframe left joins are not working as expected in pyspark: Resolved
• partitionBy in Python DataFrame reader/writer interface should not default to empty tuple: Resolved
• Add a "pretty" parameter to show: Resolved
• DataFrame Python API should work with column which has non-ascii character in it: Resolved
• In should not take Any not Column: Resolved
• Maintain binary compatibility for in function: Closed
• Good errors for invalid input to ExpectsInput expressions: Resolved
• Hide JVM stack trace for IllegalArgumentException in Python: Resolved
• Rename inSet to isin to match Pandas function: Resolved