[SPARK-2916] [MLlib] While running regression tests with dense vectors of length greater than 1000, the treeAggregate blows up after several iterations

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Component/s: MLlib, Spark Core
• Target Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Burak Yavuz
• Created: 08/Aug/14 01:54
• Updated: 16/Aug/14 05:56
• Resolved: 16/Aug/14 05:56

Description

While running any of the regression algorithms with gradient descent, the treeAggregate blows up after several iterations.

Observed on EC2 cluster with 16 nodes, matrix dimensions of 1,000,000 x 5,000

In order to replicate the problem, use aggregate multiple times, maybe over 50-60 times.

Testing lead to the possible workaround:
setting
`spark.cleaner.referenceTracking false`

seems to help. So the problem is most probably related to the cleanup.

Issue Links

• Is contained by: SPARK-3015 Removing broadcast in quick successions causes Akka timeout