[SPARK-13721] Add support for LATERAL VIEW OUTER explode()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Bogdan Raducanu
• Reporter: Ian Hellstrom
• Created: 07/Mar/16 12:13
• Updated: 28/Sep/23 07:00
• Resolved: 17/Jan/17 23:40

Description

Hive supports the LATERAL VIEW OUTER syntax to make sure that when an array is empty, the content from the outer table is still returned.

Within Spark, this is currently only possible within the HiveContext and executing HiveQL statements. It would be nice if the standard explode() DataFrame method allows the same. A possible signature would be:

explode[A, B](inputColumn: String, outputColumn: String, outer: Boolean = false)

Issue Links

• is related to: SPARK-18207 class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB
• is related to: SPARK-45369 Push down limit through generate
• relates to: SPARK-20535 R wrappers for explode_outer and posexplode_outer
• links to: [Github] Pull Request #16608 (bogdanrdc)
• links to: [Github] Pull Request #16958 (rxin)
• links to: [Github] Pull Request #17026 (bogdanrdc)