Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6404

Evenly distribute local and remote scan ranges across Impalad(s) when 100% locality is not achievable

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Distributed Exec
    • ghx-label-9

    Description

      Current scheduler tries to assign as many local reads as possible, this works well if 100% locality is achievable, but in cases where some nodes have locality while others don't an even scan ranges are assigned to the backends which results in execution skew.

      Ideally the scheduler should create an even distribution of local and remote scan ranges to avoid skew.

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-8685 Evaluate default configuration of NUM_REMOTE_EXECUTOR_CANDIDATES

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Activity

            People

              Unassigned Unassigned
              mmokhtar Mostafa Mokhtar
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: