[SPARK-25417] ArrayContains function may return incorrect result when right expression is implicitly down casted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Dilip Biswal
• Reporter: Dilip Biswal
• Created: 12/Sep/18 22:09
• Updated: 07/Jan/20 00:07
• Resolved: 20/Sep/18 12:36

Description

In ArrayContains, we currently cast the right hand side expression to match the element type of the left hand side Array. This may result in down casting and may return wrong result or questionable result.

Example :

spark-sql> select array_position(array(1), 1.34);
true

 

spark-sql> select array_position(array(1), 'foo');
null

Issue Links

• links to: [Github] Pull Request #22408 (dilipbiswal)
• links to: [Github] Pull Request #22448 (dilipbiswal)
• links to: GitHub Pull Request #22448