[SYSTEMDS-1029] Perftest: Data generator for descriptive statistics (slowdown due to update in-place)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: SystemML 0.11
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 10/Oct/16 04:11
• Updated: 13/Oct/16 16:28
• Resolved: 13/Oct/16 16:27

Description

The data generator for descriptive statistics shows severe performance issues, due to update-in-place. In order to avoid buffer pool serialization and evictions via shallow serialize, we represent sparse matrices which are subject to update in-place in CSR. This works very well for row major updates but has severe problems on column-wise updates (due to shifting of values).

There are three potential options:

• (1) Workaround: Change the data generator script from column-wise indexing to row-wise indexing and transpose after the parfor loop.
• (2) Parfor rewrite: Disable update in-place, if sparse input, no parfor update in-place, and column-wise access pattern.
• (3) Improved left indexing over CSR: Reallocate and shift the output once. This is feasible as left indexing (specifically sparse-sparse and dense-sparse) are the only operations which update the CSR representation.

Issue Links

• Is contained by: SYSTEMDS-1010 Perftest 0.11 release and related improvements