[SPARK-38864] Unpivot / melt function for Dataset API

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Enrico Minack
• Reporter: Enrico Minack
• Created: 11/Apr/22 20:06
• Updated: 27/Sep/22 12:26
• Resolved: 26/Jul/22 07:50

Description

As pointed out in SPARK-30273 and SPARK-37799, the Dataset API provides the pivot method but not its reverse operation melt. It is only provided as part of the PySpark Pandas API, implemented in Python. However, melt is not available via the PySpark DataFrame API, nor through Spark Scala Dataset API.

As mentioned in #26912, it should be implemented in Scala and invoked from PySpark.

Issue Links

• fixes: SPARK-39292 Make Dataset.melt work with struct fields
• is duplicated by: SPARK-39669 Support unpivot function
• links to: [Github] Pull Request #36150 (EnricoMi)
• links to: [Github] Pull Request #37351 (EnricoMi)
• links to: [Github] Pull Request #37351 (EnricoMi)
• links to: [Github] Pull Request #38019 (EnricoMi)