[TEZ-4027] DagAwareYarnTaskScheduler can miscompute blocked vertices and cause a hang

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.1, 0.10.0
• Fix Version/s: 0.9.2, 0.10.0
• Target Version/s: 0.9.2, 0.10.1
• Hadoop Flags: Reviewed
• Assignee: Kuhu Shukla
• Reporter: Kuhu Shukla
• Created: 14/Dec/18 21:37
• Updated: 19/Apr/19 19:21
• Resolved: 18/Dec/18 15:26

Description

In a scenario where there are retro active failures and the YARN queue is full to not allow more new container assignments, the scheduler can miscompute blocked vertex set as it tries to flip the bits upto the length of the bitset which may not be reflective of the total number of vertices. This causes no preemption and the DAG will hang.

@GuardedBy("DagAwareYarnTaskScheduler.this")
    BitSet createVertexBlockedSet() {
      BitSet blocked = new BitSet();
      Entry<Priority, RequestPriorityStats> entry = priorityStats.lastEntry();
      if (entry != null) {
        RequestPriorityStats stats = entry.getValue();
        blocked.or(stats.allowedVertices);
        blocked.flip(0, blocked.length());
        blocked.or(stats.descendants);
      }
      return blocked;
    }