[SPARK-48837] In CountVectorizer, only read binary parameter once per transform, not once per row

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 4.0.0
• Component/s: ML
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 08/Jul/24 20:33
• Updated: 09/Jul/24 03:06
• Resolved: 09/Jul/24 03:06

Description

SPARK-13629 added a binary parameter to CountVectorizer, but due to the way the code is structured the configuration parameter is read once-per-row in a UDF. Instead, we should read it once-per-transform call (similar to how other parameters are read).

Issue Links

• links to: GitHub Pull Request #47258