[SYSTEMDS-1831] Improve the efficiency of matrix subsetting

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Sprint: Sprint 3
• Assignee: Fei Hu
• Reporter: Fei Hu
• Created: 08/Aug/17 17:48
• Updated: 08/Aug/17 18:56

Description

For the rangeReIndex operation, it needs to read the whole input matrix into memory first and do the subsetting in the memory. It is not efficient because it needs to read a lot of unnecessary data, and even will be out of memory if the size of input matrix exceeds the available memory.

The plan here is to read the keys in the Hadoop sequence file, and identify the keys overlapped with the input range index. Then the values specified by the identified keys will be read.

Issue Links

• is a child of: SYSTEMDS-1809 Optimize the performance of the distributed MNIST_LeNet_Sgd model training