[SPARK-17027] PolynomialExpansion.choose is prone to integer overflow

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0, 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: ML
• External issue URL: https://stackoverflow.com/questions/38897510/spark-polynomial-expansion-vecor-size-exceeded
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 11/Aug/16 22:26
• Updated: 14/Aug/16 11:18
• Resolved: 14/Aug/16 11:01

Description

Current implementation computes power of k directly and because of that it is susceptible to integer overflow on relatively small input (4 features, degree equal 10). It would be better to use recursive formula instead.

Issue Links

• links to: [Github] Pull Request #14614 (zero323)