Major Description
During initialization, each task creates an array of objects TaskSplitMetaInfo[]. This represents unnecessary space and time overhead as each task needs only its corresponding split object. Beside the current implementation is n^2 space complexity, it leaks the inputstream.
We need to optimize that implementation by returning only a single object instead of an entire array.
rohini suggested the following:
In the vertex construct TaskSplitMetaInfo only for the split of that task instead of constructing for all splits. ie change
public static TaskSplitMetaInfo[] readSplitMetaInfo(Configuration conf, FileSystem fs) to public static TaskSplitMetaInfo getSplitMetaInfo(Configuration conf, FileSystem fs, int index) and skip reading splits below the index. If there are 1000 splits, the first task will read 1 split, second task will read 2 splits and so on instead of each task reading all the 1000 splits as is happening now.