Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10589

Improve logic of multi-node allocation

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      for (String partititon : partitions) {
       if (current++ > start) {
       break;
       }
       CandidateNodeSet<FiCaSchedulerNode> candidates =
       cs.getCandidateNodeSet(partititon);
       if (candidates == null) {
       continue;
       }
       cs.allocateContainersToNode(candidates, false);
      }

      In above logic, if we have thousands of node in one partition, we will still repeatedly access all nodes of the partition thousands of times. There is no break point where if the partition is not same for the first node, it should stop checking other nodes in that partition.

        Attachments

        1. YARN-10589-001.patch
          11 kB
          Tanu Ajmera
        2. YARN-10589-002.patch
          5 kB
          Qi Zhu
        3. YARN-10589-003.patch
          5 kB
          Qi Zhu
        4. YARN-10589-004.patch
          13 kB
          Tanu Ajmera
        5. YARN-10589-005.patch
          13 kB
          Tanu Ajmera

          Activity

            People

            • Assignee:
              tanu.ajmera Tanu Ajmera
              Reporter:
              tanu.ajmera Tanu Ajmera

              Dates

              • Created:
                Updated:

                Issue deployment