[SPARK-45926] The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Target Version/s: 3.5.1
• Assignee: Unassigned
• Reporter: Asif
• Created: 14/Nov/23 19:00
• Updated: 30/Jul/24 17:54

Description

These InMemoryV2FilterBatchScan and InMemoryBatchScan test classes are not implementing hashCode and equals correctly as they are not taking into account the pushed runtime filters. As a result they are unable to expose the TPCDS test issues which can show whether the reuse of exchange is happening correctly or not.
If these classes implement equals and hashCode taking into account the pushed runtime filters, we would see that TPCDS Q14b which should ideally be reusing the exchange containing Union , is not happening due to multiple bugs which surface in AQE.

Actual V2 DataSources like iceberg correctly implement equals and hashCode taking into account pushed runtime filters , which also expose the same issue of reuse of exchnage not happening

Issue Links

• is required by: SPARK-45866 Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )
• links to: GitHub Pull Request #43808