[SPARK-37171] Addition of forany and forall semantics to Spark Dataframes

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Dhiren Navani
• Created: 31/Oct/21 03:38
• Updated: 31/Oct/21 21:12
• Resolved: 31/Oct/21 21:12

Description

Spark dataframes currently do not have semantics of forany and forall.

Scala collections do have these semantics.

It would be nice to have them as an API as there is potential for implementing them in unoptimized form.

 

E.g. forany might be implemented as df.filter(<condition>).count > 0, which is not optimal

Issue Links

• links to: [Github] Pull Request #34450 (dhirennavani)
• links to: [Github] Pull Request #34450 (dhirennavani)