[SYSTEMDS-1025] Perftest: Large performance variability on scenario L dense (80GB)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: SystemML 0.11
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 08/Oct/16 22:48
• Updated: 10/Oct/16 23:31
• Resolved: 10/Oct/16 04:09

Description

During many runs of our entire performance testsuite, we've seen quite some performance variability, especially for scenario L dense (80GB) where spark operations are the dominating factor for end-to-end performance. These issues showed up over all algorithms and configurations but especially for multinomial classification and parfor scripts.

Let's take for example Naive Bayes over the dense 10M x 1K input with 20 classes. Below are the results of 7 consecutive runs:

NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 67
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 362
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 484
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 64
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 310
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 91
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 68

After a detailed investigation, it seems that imbalance, garbage collection, and poor data locality are the reasons:

• First, we generated the inputs with our Spark backend. Apparently, the rand operation caused imbalance due to garbage collection of some nodes. However, this is a very realistic scenario as we cannot always assume perfect balance.
• Second, especially for multinomial classification and parfor scripts, the intermediates are not just vectors but larger matrices or there are simply more intermediates. This led again to more garbage collection.
• Third, the scheduler delay of 3s for pending tasks was exceeded due to garbage collection, leading to remote execution which significantly slowed down the overall execution.

To resolve these issues, we should make the following two changes:

• (1) More conservative configuration of spark.locality.wait in systemml's preferred spark configuration, where we did not consider this at all so far.
• (2) Improvements of reduce-all operations which current unnecessarily create intermediate pair outputs and hence unnecessary Tuple2 and MatrixIndexes objects.

With a default scheduler delay of 5s instead of the default 3s as well as improved reduce-all for mapmm, groupedagg, tsmm, tsmm2, zipmm, and uagg, we got the following promising results (which include spark context creation and initial read):

NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 52
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 45
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 44
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 44
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 51
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 50
NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 47

cc reinwald niketanpansare freiss

Issue Links

• Is contained by: SYSTEMDS-1010 Perftest 0.11 release and related improvements