[CARBONDATA-3288] Implement Distributed Index Cache Server

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.3
• Assignee: Unassigned
• Reporter: Kunal Kapoor
• Created: 05/Feb/19 11:06
• Updated: 05/Apr/19 06:58

Description

Implement a Distributed index cache server which can prune and cache block/blocklet datamaps to reduce the memory footprint on the carbon driver application.

 

Design Document:

https://docs.google.com/document/d/161NXxrKLPucIExkWip5mX00x2iOPH6bvsuQnCzzp47E/edit?ts=5c542ab4# 

Sub-Tasks

• Implement a DistributableIndexPruneRDD and IndexPruneFileFormat: Resolved
• Implement a Hadoop RPC framwork for communication: Resolved
• Display original query in Indexserver Job: Resolved
• Large response size Exception is thrown from index server.: Resolved
• Make use of LRU mandatory when using IndexServer: Resolved
• Implement Show Cache for IndexServer and MV: Closed
• Implement Executor ID based task distribution for Index Server: Resolved
• Expose a DDL to add index size and data size to tableStatus for legacy segments: Resolved
• Optimize the performance of select count(*) for index server: Resolved
• Fixed id based distribution for show cache command: Resolved
• Add usage and deployment document for index server: Resolved