Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1492 truly shared cache for jars (jobjar/libjar)
  3. YARN-3637

Handle localization sym-linking correctly at the YARN level

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha4
    • None
    • None
    • Reviewed

    Description

      The shared cache needs to handle resource sym-linking at the YARN layer. Currently, we let the application layer (i.e. mapreduce) handle this, but it is probably better for all applications if it is handled transparently.

      Here is the scenario:
      Imagine two separate jars (with unique checksums) that have the same name job.jar.

      They are stored in the shared cache as two separate resources:
      checksum1/job.jar
      checksum2/job.jar

      A new application tries to use both of these resources, but internally refers to them as different names:
      foo.jar maps to checksum1
      bar.jar maps to checksum2

      When the shared cache returns the path to the resources, both resources are named the same (i.e. job.jar). Because of this, when the resources are localized one of them clobbers the other. This is because both symlinks in the container_id directory are the same name (i.e. job.jar) even though they point to two separate resource directories.

      Originally we tackled this in the MapReduce client by using the fragment portion of the resource url. This, however, seems like something that should be solved at the YARN layer.

      Attachments

        1. YARN-3637-trunk.001.patch
          6 kB
          Chris Trezzo
        2. YARN-3637-trunk.002.patch
          6 kB
          Chris Trezzo
        3. YARN-3637-trunk.003.patch
          8 kB
          Chris Trezzo

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ctrezzo Chris Trezzo
            ctrezzo Chris Trezzo
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment