Major Description
In some scenario, the input amount of loading data is small, but carbondata still distribute them to each executors (nodes) to do local-sort, thus resulting to small carbondata files generated by each executor.
In some extreme conditions, if the cluster is big enough or if the amount of data is small enough, the carbondata file contains only one blocklet or page.
I think a new strategy should be introduced to solve the above problem.
The new strategy should:
- be able to control the minimum amount of input data for each node
- ignore data locality otherwise it may always choose a small portion of particular nodes
Attachments
Issue Links
- links to
