[SPARK-18368] Regular expression replace throws NullPointerException when serialized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.1, 2.1.0
• Fix Version/s: 2.0.3, 2.1.0
• Component/s: SQL
• Target Version/s: 2.0.3, 2.1.0
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 08/Nov/16 22:56
• Updated: 23/May/17 10:54
• Resolved: 09/Nov/16 19:02

Description

This query fails with a NullPointerException on line 247:

SELECT POSEXPLODE(SPLIT(REGEXP_REPLACE(ranks, '[\\[ \\]]', ''), ',')) AS (rank, col0) FROM table;

The problem is that POSEXPLODE is causing the REGEXP_REPLACE to be serialized after it is instantiated. The null value is a transient StringBuffer that should hold the result. The fix is to make the result value lazy.

Issue Links

• is duplicated by: SPARK-13819 using a regexp_replace in a group by clause raises a nullpointerexception
• relates to: SPARK-18387 Test that expressions can be serialized
• links to: [Github] Pull Request #15816 (rdblue)
• links to: [Github] Pull Request #15834 (rdblue)
• links to: [Github] Pull Request #15847 (rdblue)