Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
2.2.0
-
None
-
None
Description
In CartesianRDD, group each iterator to multiple groups. Thus in the second iteration, the data with be fetched (num of data)/groupSize times, rather than (num of data) times.
The test results are:
Test Environment : 3 workers, each has 10 cores, 30G memory, 1 executor
Test data : users : 480,189, each is a 10-dim vector, and items : 17770, each is a 10-dim vector.
With default CartesianRDD, cartesian time is 2420.7s.
With this proposal, cartesian time is 45.3s
50x faster than the original method.
