Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6115

Investigate scheduling options for block metadata loading threads

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.11.0
    • Fix Version/s: None
    • Component/s: Catalog
    • Epic Color:
      ghx-label-4

      Description

      In the current implementation of IMPALA-5429, we spin up an executor pool of threads every time we load the table and close it once the load has finished/failed. We need to investigate the following options and see if they can give better throughput.

      1. Single thread pool for the process lifetime to minimize the startup and teardown latency and all the block metadata loading tasks are submitted to it.
      2. Clustered scheduling of threads rather than submitting them all at once.
      3. Any other scheduling options that guarantee fairness to all the tables and are more performant.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bharathv bharath v
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: