Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2841

admission control always estimate memory by using all available backend even for single node plan.

    XMLWordPrintableJSON

Details

    Description

      Admission control always estimates query total memory by using all available backends, even for single node plan.

      void SimpleScheduler::UpdateMembership(
          const StatestoreSubscriber::TopicDeltaMap& incoming_topic_deltas,
          vector<TTopicDelta>* subscriber_topic_updates) {
      ...
      if (metrics_ != NULL) num_backends_metric_->set_value(current_membership_.size());
      ...
      }
      

      schedule->set_num_hosts(max<int64_t>(num_backends_metric_->value(), 1));

      int64_t QuerySchedule::GetClusterMemoryEstimate() const {
        DCHECK_GT(num_hosts_, 0);
        const int64_t total_cluster_mem = GetPerHostMemoryEstimate() * num_hosts_;
        DCHECK_GE(total_cluster_mem, 0); // Assume total cluster memory fits in an int64_t.
        return total_cluster_mem;
      }
      
      Status AdmissionController::CanAdmitRequest(const string& pool_name,
          const int64_t max_requests, const int64_t mem_limit, const QuerySchedule& schedule,
          bool admit_from_queue) {
      const int64_t query_total_estimated_mem = schedule.GetClusterMemoryEstimate();
      

      Especially on large cluster, this cause memory estimation far off for trivial queries (without stats).

      Attachments

        Activity

          People

            mjacobs Matthew Jacobs
            jyu@cloudera.com Juan Yu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: