[SYSTEMDS-677] Random data generator for decision tree fails w/ data type mismatch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemML 0.9
• Fix Version/s: SystemML 0.10
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 10/May/16 02:10
• Updated: 10/May/16 18:49
• Resolved: 10/May/16 18:49

Description

The data generator for decision tree is composed of a shell script that calls two dml scripts in order to apply the file-based transform (which requires an existing file during compilation) in the second script. However, there is a data type mismatch as the first script outputs a matrix and the second script expects a frame.

This task covers (1) a script level change to output a frame from the first script, and (2) a fix for writing the frame meta data file with a value type accepted by the subsequent transform.

Note that the script level change already exploits matrix-frame casting which has been introduced as part of SYSTEMML-554 but this builtin function is as of today only supported in CP. This means, the data generator only works for small data that fits into the driver memory. Once the Spark/MR converters from SYSTEMML-560 are fully integrated, the script will runs for large data too without further script changes.

Issue Links

• relates to: SYSTEMDS-554 Frame data type support