[YARN-9833] Race condition when DirectoryCollection.checkDirs() runs during container launch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.2, 3.1.4, 2.10.2
• Target Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 13/Sep/19 04:08
• Updated: 05/Mar/21 15:37
• Resolved: 17/Sep/19 14:12

Description

During endurance testing, we found a race condition that cause an empty localDirs being passed to container-executor.

The problem is that DirectoryCollection.checkDirs() clears three collections:

    this.writeLock.lock();
    try {
      localDirs.clear();
      errorDirs.clear();
      fullDirs.clear();
      ...

This happens in critical section guarded by a write lock. When we start a container, we retrieve the local dirs by calling dirsHandler.getLocalDirs(); which in turn invokes DirectoryCollection.getGoodDirs(). The implementation of this method is:

List<String> getGoodDirs() {
    this.readLock.lock();
    try {
      return Collections.unmodifiableList(localDirs);
    } finally {
      this.readLock.unlock();
    }
  }

So we're also in a critical section guarded by the lock. But Collections.unmodifiableList() only returns a view of the collection, not a copy. After we get the view, MonitoringTimerTask.run() might be scheduled to run and immediately clears localDirs.
This caused a weird behaviour in container-executor, which exited with error code 35 (COULD_NOT_CREATE_WORK_DIRECTORIES).

Therefore we can't just return a view, we must return a copy with ImmutableList.copyOf().

Credits to Szilard Nemeth for analyzing and determining the root cause.

Issue Links

• is duplicated by: YARN-8786 LinuxContainerExecutor fails sporadically in create_local_dirs
• is related to: YARN-8786 LinuxContainerExecutor fails sporadically in create_local_dirs
• relates to: YARN-10562 Follow up changes for YARN-9833