Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7282 Shared Cache Phase 2
  3. YARN-5727

Improve YARN shared cache support for LinuxContainerExecutor



    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None


      When running LinuxContainerExecutor in a secure mode (yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users set to false), all localized files are owned by the user that owns the container which localized the resource. This presents a problem for the shared cache when a YARN application requests a resource to be uploaded to the shared cache that has a non-public visibility. The shared cache uploader (running as the node manager user) does not have access to the localized files and can not compute the checksum of the file or upload it to the cache. The solution should ideally satisfy the following three requirements:

      1. Localized files should still be safe/secure. Other users that run containers should not be able to modify, or delete the publicly localized files of others.
      2. The node manager user should be able to access these files for the purpose of checksumming and uploading to the shared cache without being a privileged user.
      3. The solution should avoid making unnecessary copies of the localized files.


        1. YARN-5727.001.patch
          183 kB
          zhenzhao wang
        2. YARN-5727-Design-v1.pdf
          72 kB
          Chris Trezzo
        3. YARN-5727-Design-v2.pdf
          157 kB
          zhenzhao wang

        Issue Links



              wzzdreamer zhenzhao wang
              ctrezzo Chris Trezzo
              0 Vote for this issue
              10 Start watching this issue