[SYSTEMDS-2031] Perftest: Unnecessary compression of incompressible blocks

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.0.0
• Epic Link: Perftest and QA for SystemML 1.0 release
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 30/Nov/17 03:32
• Updated: 30/Nov/17 22:27
• Resolved: 30/Nov/17 21:50

Description

By default, we apply compression for data sets that are known to exceed aggregate memory if all operations of the given script that touch the respective input are supported over compressed matrices.

On the perftest 800GB dense scenario, this leads to a slight slowdown and increase in the matrix size due to incompressible data, where each block is represented as follows:

--col groups sizes (OLE,RLE,DDC1,DDC2,UC): 0,0,0,0,1000
--compression ratio: 0.999475777837746

We should investigate the set of incompressible columns as well as final representation and simply return the uncompressed block in such such scenarios.

Issue Links

• is related to: SYSTEMDS-1916 Improve CLA robustness for incompressible inputs