[SPARK-19454] Improve DataFrame.replace API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.6.0, 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.2.0
• Component/s: PySpark, SQL
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 03/Feb/17 20:07
• Updated: 06/Feb/18 09:48
• Resolved: 05/Apr/17 18:51

Description

Current implementation suffers from following issues:

• It is possible to use dict as to_replace, but we cannot skip or use None as the value value (although it is ignored). This requires passing "magic" values:

  df = sc.parallelize([("Alice", 1, 3.0)]).toDF()
  df.replace({"Alice": "Bob"}, 1)
  

• Code doesn't check if provided types are correct. This can lead to exception in Py4j (harder to diagnose):

   df.replace({"Alice": 1}, 1)
  

  or silent failures (with bundled Py4j version):

   df.replace({1: 2, 3.0: 4.1, "a": "b"}, 1)
  

Issue Links

• is related to: SPARK-19453 Correct DataFrame.replace docs
• links to: [Github] Pull Request #16793 (zero323)