[SYSTEMDS-2020] Perftest bivariate statistics w/ codegen causes unnecessary sparse-dense conversions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.0.0
• Epic Link: Perftest and QA for SystemML 1.0 release
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 18/Nov/17 00:10
• Updated: 19/Nov/17 21:55
• Resolved: 19/Nov/17 21:55

Description

[8,1] scale-categorical
[3,1] scale-categorical
[10,1] scale-categorical
[6,1] scale-categorical
[4,1] scale-categorical
[1,1] scale-categorical
[2,1] scale-categorical
[7,1] scale-categorical
[5,1] scale-categorical
[9,1] scale-categorical
[13,1] scale-scale
[13,2] scale-categorical
[14,1] scale-scale
[11,1] scale-scale
[14,2] scale-categorical
[12,1] scale-scale
[12,2] scale-categorical
[11,2] scale-categorical
[15,1] scale-scale
[16,1] scale-scale
[15,2] scale-categorical
[16,2] scale-categorical
[11,3] scale-categorical
[12,3] scale-categorical
[15,3] scale-categorical
[12,4] scale-categorical
[14,3] scale-categorical
[16,3] scale-categorical
[13,3] scale-categorical
[15,4] scale-categorical
[12,5] scale-categorical
[16,4] scale-categorical
[13,4] scale-categorical
[15,5] scale-categorical
[11,4] scale-categorical
[12,6] scale-categorical
[13,5] scale-categorical
[16,5] scale-categorical
[14,4] scale-categorical
[15,6] scale-categorical
[11,5] scale-categorical
[12,7] scale-categorical
[13,6] scale-categorical
[14,5] scale-categorical
[16,6] scale-categorical
[12,8] scale-categorical
[13,7] scale-categorical
[15,7] scale-categorical
[14,6] scale-categorical
[11,6] scale-categorical
[16,7] scale-categorical
[12,9] scale-categorical
[13,8] scale-categorical
[14,7] scale-categorical
[11,7] scale-categorical
[16,8] scale-categorical
[14,8] scale-categorical
[13,9] scale-categorical
[12,10] scale-categorical
[11,8] scale-categorical
[15,8] scale-categorical
[16,9] scale-categorical
[14,9] scale-categorical
[13,10] scale-categorical
[11,9] scale-categorical
[15,9] scale-categorical
[16,10] scale-categorical
[14,10] scale-categorical
[11,10] scale-categorical
[15,10] scale-categorical
[19,1] scale-scale
[17,1] scale-scale
[19,2] scale-categorical
[17,2] scale-categorical
[18,1] scale-scale
[18,2] scale-categorical
[19,3] scale-categorical
[20,1] scale-scale
[17,3] scale-categorical
[20,2] scale-categorical
[18,3] scale-categorical
[19,4] scale-categorical
[20,3] scale-categorical
[17,4] scale-categorical
[19,5] scale-categorical
[18,4] scale-categorical
[17,5] scale-categorical
[19,6] scale-categorical
[20,4] scale-categorical
[18,5] scale-categorical
[17,6] scale-categorical
[19,7] scale-categorical
[20,5] scale-categorical
[18,6] scale-categorical
[17,7] scale-categorical
[19,8] scale-categorical
[20,6] scale-categorical
[18,7] scale-categorical
[17,8] scale-categorical
[19,9] scale-categorical
[17,9] scale-categorical
[18,8] scale-categorical
[20,7] scale-categorical
[19,10] scale-categorical
[18,9] scale-categorical
[20,8] scale-categorical
[17,10] scale-categorical
[20,9] scale-categorical
[18,10] scale-categorical
[20,10] scale-categorical
[8,2] categorical-categorical
[3,2] categorical-categorical
[4,2] categorical-categorical
[10,2] categorical-categorical
[9,2] categorical-categorical
[6,2] categorical-categorical
[2,2] categorical-categorical
[1,2] categorical-categorical
[7,2] categorical-categorical
[5,2] categorical-categorical
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP43: Converted 566x640, nnz=9829 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP22: Converted 727x640, nnz=9886 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x640, nnz=9934 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP50: Converted 1100x640, nnz=9934 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP79: Converted 1100x640, nnz=9922 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP64: Converted 1100x640, nnz=9941 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP36: Converted 1051x640, nnz=9924 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP29: Converted 870x640, nnz=9916 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP71: Converted 1100x640, nnz=9923 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x640, nnz=9938 sideways input matrix from sparse to dense.
[1,2] ordinal-ordinal
[9,3] categorical-categorical
[2,2] ordinal-ordinal
[8,3] categorical-categorical
[10,3] categorical-categorical
[5,2] ordinal-ordinal
[6,3] categorical-categorical
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
[7,3] categorical-categorical
[4,2] ordinal-ordinal
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9935 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x1100, nnz=9918 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
[3,2] ordinal-ordinal
[9,4] categorical-categorical
[8,4] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x429, nnz=9790 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x429, nnz=9849 sideways input matrix from sparse to dense.
[6,4] categorical-categorical
[10,4] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9900 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9904 sideways input matrix from sparse to dense.
[7,4] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x429, nnz=9915 sideways input matrix from sparse to dense.
[9,5] categorical-categorical
[8,5] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x635, nnz=9889 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x635, nnz=9848 sideways input matrix from sparse to dense.
[10,5] categorical-categorical
[6,5] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9922 sideways input matrix from sparse to dense.
[7,5] categorical-categorical
[1,3] categorical-categorical
[2,3] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9930 sideways input matrix from sparse to dense.
[5,3] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
[9,6] categorical-categorical
[8,6] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9963 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9969 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9926 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9930 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9940 sideways input matrix from sparse to dense.
[10,6] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
[8,7] categorical-categorical
[7,6] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9915 sideways input matrix from sparse to dense.
[6,6] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9966 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
[9,7] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9928 sideways input matrix from sparse to dense.
[3,3] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x1100, nnz=9957 sideways input matrix from sparse to dense.
[8,8] categorical-categorical
[4,3] categorical-categorical
[9,8] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x1100, nnz=9915 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9961 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9933 sideways input matrix from sparse to dense.
[10,7] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=1100 sideways input matrix from sparse to dense.
[6,7] categorical-categorical
[3,3] ordinal-ordinal
[7,7] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9949 sideways input matrix from sparse to dense.
[1,3] ordinal-ordinal
[8,9] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9922 sideways input matrix from sparse to dense.
[2,3] ordinal-ordinal
[5,3] ordinal-ordinal
[9,9] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 727x1100, nnz=9940 sideways input matrix from sparse to dense.
[8,10] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x444, nnz=9795 sideways input matrix from sparse to dense.
[3,4] categorical-categorical
[6,8] categorical-categorical
[10,8] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x429, nnz=9883 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9954 sideways input matrix from sparse to dense.
[9,10] categorical-categorical
[4,3] ordinal-ordinal
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 727x444, nnz=9830 sideways input matrix from sparse to dense.
[3,4] ordinal-ordinal
[7,8] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
[6,9] categorical-categorical
[10,9] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9957 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9963 sideways input matrix from sparse to dense.
[7,9] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
[10,10] categorical-categorical
[6,10] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9900 sideways input matrix from sparse to dense.
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9902 sideways input matrix from sparse to dense.
[7,10] categorical-categorical
17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x444, nnz=9885 sideways input matrix from sparse to dense.
[3,5] categorical-categorical
[1,4] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x635, nnz=9922 sideways input matrix from sparse to dense.
[5,4] categorical-categorical
[4,4] categorical-categorical
[2,4] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9905 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9888 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x429, nnz=9890 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9881 sideways input matrix from sparse to dense.
[5,4] ordinal-ordinal
[1,4] ordinal-ordinal
[2,4] ordinal-ordinal
[4,4] ordinal-ordinal
[3,5] ordinal-ordinal
[5,5] categorical-categorical
[1,5] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9920 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x635, nnz=9933 sideways input matrix from sparse to dense.
[2,5] categorical-categorical
[3,6] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9932 sideways input matrix from sparse to dense.
[4,5] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9953 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x635, nnz=9939 sideways input matrix from sparse to dense.
[5,5] ordinal-ordinal
[1,5] ordinal-ordinal
[2,5] ordinal-ordinal
[4,5] ordinal-ordinal
[3,7] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9950 sideways input matrix from sparse to dense.
[4,6] categorical-categorical
[5,6] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9943 sideways input matrix from sparse to dense.
[2,6] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
[1,6] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9955 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9962 sideways input matrix from sparse to dense.
[3,8] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9959 sideways input matrix from sparse to dense.
[1,7] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
[4,7] categorical-categorical
[3,9] categorical-categorical
[5,7] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9955 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9949 sideways input matrix from sparse to dense.
[2,7] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x1100, nnz=9940 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9962 sideways input matrix from sparse to dense.
[1,8] categorical-categorical
[3,10] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9952 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x444, nnz=9867 sideways input matrix from sparse to dense.
[4,8] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x1100, nnz=9952 sideways input matrix from sparse to dense.
[2,8] categorical-categorical
[5,8] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9952 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
[1,9] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
[4,9] categorical-categorical
[5,9] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x1100, nnz=9962 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
[2,9] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
[1,10] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9906 sideways input matrix from sparse to dense.
[5,10] categorical-categorical
[4,10] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x444, nnz=9895 sideways input matrix from sparse to dense.
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x444, nnz=9895 sideways input matrix from sparse to dense.
[2,10] categorical-categorical
17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9887 sideways input matrix from sparse to dense.

Issue Links

• relates to: SYSTEMDS-1284 Code generation for operator fusion