Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4440

FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      It seems there is a bug on FSAppAttempt#getAllowedLocalityLevelByTime method

      // default level is NODE_LOCAL
          if (! allowedLocalityLevel.containsKey(priority)) {
            allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
            return NodeType.NODE_LOCAL;
          }
      

      If you first invoke this method, it doesn't init time in lastScheduledContainer and this will lead to execute these code for next invokation:

          // check waiting time
          long waitTime = currentTimeMs;
          if (lastScheduledContainer.containsKey(priority)) {
            waitTime -= lastScheduledContainer.get(priority);
          } else {
            waitTime -= getStartTime();
          }
      

      the waitTime will subtract to FsApp startTime, and this will be easily more than the delay time and allowedLocality degrade. Because FsApp startTime will be start earlier than currentTimeMs. So we should add the initial time of priority to prevent comparing with FsApp startTime and allowedLocalityLevel degrade. And this problem will have more negative influence for small-jobs. The YARN-4399 also discuss some problem in aspect of locality.

        Attachments

        1. YARN-4440.003.patch
          5 kB
          Yiqun Lin
        2. YARN-4440.002.patch
          5 kB
          Yiqun Lin
        3. YARN-4440.001.patch
          5 kB
          Yiqun Lin

          Issue Links

            Activity

              People

              • Assignee:
                linyiqun Yiqun Lin
                Reporter:
                linyiqun Yiqun Lin
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: