Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8784

DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0, 3.1.1
    • Fix Version/s: 3.2.0, 3.1.2
    • Component/s: nodemanager
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      DockerLinuxContainerRuntime bind mounts the filecache and usercache directories into the container to allow tasks to access entries in the distributed cache. However it only bind mounts directories on disks that are considered good, and disks that are full or bad are not in that list. If a container tries to run with a distributed cache entry that has been previously localized to a disk that is now considered full/bad, the dist cache directory will not be bind-mounted into the container's filesystem namespace. At that point any symlinks in the container's current working directory that point to those disks will reference invalid paths.

        Attachments

        1. YARN-8784.001.patch
          7 kB
          Eric Badger

          Activity

            People

            • Assignee:
              ebadger Eric Badger
              Reporter:
              jlowe Jason Darrell Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: