Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561

[Umbrella ticket] Performance issues in YARN+MR

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: mrv2, performance
    • Labels:
      None

      Description

      Been working on measuring performance of YARN+MR relative to the 0.20.xx release line together with Karam Singh.

      This is an umbrella ticket to track all the issues related to performance.

        Attachments

        1.
        		 Shuffle benchmark is nearly 1.5x slower in 0.23 Sub-task Closed Vinod Kumar Vavilapalli Actions
        2.
        		 Scan benchmark is more than 1.5x slower in 0.23 Sub-task Resolved Unassigned Actions
        3.
        		 AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly Sub-task Closed Vinod Kumar Vavilapalli Actions
        4.
        		 Batch jobHistory disk flushes Sub-task Closed Siddharth Seth Actions
        5.
        		 Counters occupy a good part of AM heap Sub-task Closed Vinod Kumar Vavilapalli Actions
        6.
        		 Optimize YARN API calls Sub-task Resolved Vinod Kumar Vavilapalli Actions
        7.
        		 MR AM slows down due to repeatedly constructing ContainerLaunchContext Sub-task Closed Vinod Kumar Vavilapalli Actions
        8.
        		 Extraneous JobConf objects in AM heap Sub-task Closed Vinod Kumar Vavilapalli Actions
        9.
        		 Optimize Job's progress calculations in MR AM Sub-task Closed Vinod Kumar Vavilapalli Actions
        10.
        		 TaskAttemptListener holds a global lock for all task-updates Sub-task Closed Vinod Kumar Vavilapalli Actions
        11.
        		 MR AM's dispatcher is blocked by heartbeats to ResourceManager Sub-task Closed Vinod Kumar Vavilapalli Actions
        12.
        		 Compression benchmark run-time increased by 13% in 0.23 Sub-task Closed Vinod Kumar Vavilapalli Actions
        13.
        		 ContainerLocalizer should request new resources after completing the current one Sub-task Closed Siddharth Seth Actions
        14.
        		 Thread pool for launching containers in MR AM not expanding as expected Sub-task Closed Vinod Kumar Vavilapalli Actions
        15.
        		 TaskHeartbeatHandler holds a global lock for all task-updates Sub-task Closed Siddharth Seth Actions
        16.
        		 CapacityScheduler should be more conservative assigning off-switch requests Sub-task Closed Arun Murthy Actions
        17.
        		 DFSIO read throughput is decreased by 16% in 0.23.1 than Hadoop-0.20.204 on 350 nodes size cluster. Sub-task Resolved Vinod Kumar Vavilapalli Actions
        18.
        		 AM-NM RPC calls occasionally takes a long time to respond Sub-task Open Unassigned Actions
        19.
        		 Default AM heartbeat interval should be one second Sub-task Closed Hitesh Shah Actions
        20.
        		 Make gridmix performance on YARN+MR to match or exceed that on 1.0 Sub-task Resolved Unassigned Actions
        21.
        		 Tasks may take upto 3 seconds to exit after completion Sub-task Closed Siddharth Seth Actions
        22.
        		 MR AM's ContainerAllocator is assigning the allocated containers very slowly Sub-task Closed Vinod Kumar Vavilapalli Actions
        23.
        		 Lower default allocation sizes, fix allocation configurations and document them Sub-task Closed Harsh J Actions
        24.
        		 RackResolver should maintain a cache to avoid repetitive lookups. Sub-task Closed Vinod Kumar Vavilapalli Actions
        25.
        		 Data Locality suffers if the AM asks for containers using IPs instead of hostnames Sub-task Closed Siddharth Seth Actions
        26.
        		 Counters are getting calculated twice at job-finish and delaying clients. Sub-task Closed Vinod Kumar Vavilapalli Actions
        27.
        		 Counters aggregation slowed down significantly after MAPREDUCE-3749 Sub-task Closed Vinod Kumar Vavilapalli Actions
        28.
        		 MapReduce job submission time has increased in 0.23 when compared to 0.20.206 Sub-task Open Unassigned Actions
        29.
        		 RM scalability runtime is worse than 0.20.204 by 14.2% Sub-task Resolved Unassigned Actions

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              vinodkv Vinod Kumar Vavilapalli
              Reporter:
              vinodkv Vinod Kumar Vavilapalli

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment