[SPARK-16275] Implement all the Hive fallback functions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 29/Jun/16 04:42
• Updated: 09/Sep/19 16:30
• Resolved: 09/Sep/19 16:30

Description

As of Spark 2.0, Spark falls back to Hive for only the following built-in functions:

    "elt", "hash", "java_method", "histogram_numeric",
    "map_keys", "map_values",
    "parse_url", "percentile", "percentile_approx", "reflect", "sentences", "stack", "str_to_map",
    "xpath", "xpath_boolean", "xpath_double", "xpath_float", "xpath_int", "xpath_long",
    "xpath_number", "xpath_short", "xpath_string",

    // table generating function
    "inline", "posexplode"

The goal of the ticket is to implement all of these in Spark so we don't need to fall back into Hive's UDFs.

Issue Links

• contains: SPARK-16270 Implement xpath user defined functions

Sub-Tasks

• Implement elt SQL function: Resolved
• Implement java_method SQL function: Closed
• Implement map_keys SQL function: Resolved
• Implement map_values SQL function: Resolved
• Implement histogram_numeric SQL function: Resolved
• Implement parse_url SQL function: Resolved
• Implement percentile SQL function: Resolved
• Implement percentile_approx SQL function: Resolved
• Implement reflect and java_method SQL function: Resolved
• Implement sentences SQL function: Resolved
• Implement stack table generating function: Resolved
• Implement str_to_map SQL function: Resolved
• Implement inline table generating function: Resolved
• Implement posexplode table generating function: Resolved
• Implement code generation for elt function: Closed
• Implement code generation for parse_url function: Closed
• checkEvaluation should support NaN values: Resolved
• Moves QuantileSummaries to project catalyst from sql so that it can be used to implement percentile_approx: Resolved
• Alias percentile_approx approx_percentile: Resolved
• Implement NGrams aggregate function: Resolved