Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561 [Umbrella ticket] Performance issues in YARN+MR
  3. MAPREDUCE-3399

ContainerLocalizer should request new resources after completing the current one

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.23.0
    • 0.23.1
    • mrv2, nodemanager
    • None
    • Reviewed
    • Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second.

    Description

      Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.

      Attachments

        1. MR3399_v2.txt
          5 kB
          Siddharth Seth
        2. MR3399_v1.txt
          3 kB
          Siddharth Seth

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment