Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6690

Limit the number of resources a single map reduce job can submit for localization

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha1
    • None
    • None
    • Reviewed


      Users will sometimes submit a large amount of resources to be localized as part of a single map reduce job. This can cause issues with YARN localization that destabilize the cluster and potentially impact other user jobs. These resources are specified via the files, libjars, archives and jobjar command line arguments or directly through the configuration (i.e. distributed cache api). The resources specified could be too large in multiple dimensions:

      1. Total size
      2. Number of files
      3. Size of an individual resource (i.e. a large fat jar)

      We would like to encourage good behavior on the client side by having the option of enforcing resource limits along the above dimensions.

      There should be a separate effort to enforce limits at the YARN layer on the server side, but this jira is only covering the map reduce layer on the client side. In practice, having these client side limits will get us a long way towards preventing these localization anti-patterns.


        1. MAPREDUCE-6690-trunk-v1.patch
          17 kB
          Chris Trezzo
        2. MAPREDUCE-6690-trunk-v2.patch
          20 kB
          Chris Trezzo
        3. MAPREDUCE-6690-trunk-v3.patch
          31 kB
          Chris Trezzo
        4. MAPREDUCE-6690-trunk-v4.patch
          32 kB
          Chris Trezzo
        5. MAPREDUCE-6690-trunk-v5.patch
          31 kB
          Chris Trezzo
        6. MAPREDUCE-6690-trunk-v6.patch
          34 kB
          Chris Trezzo
        7. MAPREDUCE-6690-trunk-v7.patch
          37 kB
          Chris Trezzo

        Issue Links


          This comment will be Viewable by All Users Viewable by All Users


            ctrezzo Chris Trezzo
            ctrezzo Chris Trezzo
            0 Vote for this issue
            8 Start watching this issue




                Issue deployment