[SPARK-8159] Improve expression function coverage (Spark 1.5)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Epic Link: Tungsten Phase 1
• Sprint: Spark 1.5 release
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 08/Jun/15 07:41
• Updated: 13/Aug/15 05:15
• Resolved: 05/Aug/15 05:42

Description

This is an umbrella ticket to track new expressions we are adding to SQL/DataFrame.

For each new expression, we should:
1. Add a new Expression implementation in org.apache.spark.sql.catalyst.expressions
2. If applicable, implement the code generated version (by implementing genCode).
3. Add comprehensive unit tests (for all the data types the expressions support).
4. If applicable, add a new function for DataFrame in org.apache.spark.sql.functions, and python/pyspark/sql/functions.py for Python.

For date/time functions, put them in expressions/datetime.scala, and create a DateTimeFunctionSuite.scala for testing.

Issue Links

• duplicates: SPARK-9845 Add built-in UDF
• is blocked by: SPARK-7285 Audit missing Hive functions
• is related to: SPARK-7075 Project Tungsten (Spark 1.5 Phase 1)
• is related to: SPARK-8829 Improve expression performance
• relates to: SPARK-9571 Improve expression function coverage