In the current implementation of
IMPALA-5429, we spin up an executor pool of threads every time we load the table and close it once the load has finished/failed. We need to investigate the following options and see if they can give better throughput.
1. Single thread pool for the process lifetime to minimize the startup and teardown latency and all the block metadata loading tasks are submitted to it.
2. Clustered scheduling of threads rather than submitting them all at once.
3. Any other scheduling options that guarantee fairness to all the tables and are more performant.