Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6115

Investigate scheduling options for block metadata loading threads

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • Impala 2.11.0
    • None
    • Catalog
    • ghx-label-4

    Description

      In the current implementation of IMPALA-5429, we spin up an executor pool of threads every time we load the table and close it once the load has finished/failed. We need to investigate the following options and see if they can give better throughput.

      1. Single thread pool for the process lifetime to minimize the startup and teardown latency and all the block metadata loading tasks are submitted to it.
      2. Clustered scheduling of threads rather than submitting them all at once.
      3. Any other scheduling options that guarantee fairness to all the tables and are more performant.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5429 Use a thread pool to load block metadata in parallel

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              bharathv Bharath Vissapragada
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: