[SPARK-37055] Apply 'compute.eager_check' across all the codebase

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: dch nguyen
• Shepherd: hyukjin.kwon
• Created: 19/Oct/21 06:00
• Updated: 12/Dec/22 18:11

Description

As hyukjin.kwon guide

1 Make every input validation like this covered by the new configuration. For example:

- a == b
+ def eager_check(f): # Utility function 
+ return not config.compute.eager_check and f() 
+ 
+ eager_check(lambda: a == b)

2 We should check if the output makes sense although the behaviour is not matched with pandas'. If the output does not make sense, we shouldn't cover it with this configuration.

3 Make this configuration enabled by default so we match the behaviour to pandas' by default.

 

We have to make sure listing which API is affected in the description of 'compute.eager_check'

Issue Links

• is a parent of: SPARK-36968 ps.Series.dot raise "matrices are not aligned" if index is not same
• relates to: SPARK-37002 Introduce the 'compute.eager_check' option

Sub-Tasks

• Skip check monotonic increasing for Series.asof with 'compute.eager_check': Resolved
• Skip hasnans check in numops if eager_check disable: Resolved
• Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled: Resolved
• Skip equal check in LocIndexer._select_cols_by_iterable if eager_check disable: In Progress