Major Description
Problem:
When number of splits are high, index server performance is slow as compared to old flow(Driver caching). This is because data is transferred over network is more and causing performance bottleneck.
Solution:
- If data transferred is less we can sent through network, but when it grows we can write to file and only send file name and in Main driver it will read the file and construct input split.
- Use snappy to compress the data, so data transferred through network/written to file size will be less, so IO time wont impact performance
- In main driver pruning is done in multiple thread, added same for index executor as now index executor will do the pruning
- In case of block cache no need to send blockletdetailinfo object as size is more and same can be constructed in executor from file footer
Attachments
Issue Links
- links to
