Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9255

Avoid unnecessary concurrent resource consumption by fragments.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Frontend
    • Labels:

      Description

      In Planner.computeResourceReqs(), we assume that all fragment's peak resource consumption occurs at the same time, resulting in higher than necessary estimates. There are a few things that need to change here to address this inefficiency:

      • We should consider changing the backend logic so that fragments don't start up and claim resources until their dependencies are ready.
      • We need to change the backend logic so that resources can be released when needed, e.g. the join build fragment's thread should exit once the build is done, instead of waiting for the probe threads to all exit
      • We need to update the planner to reflect the new behaviour.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: