[MAPREDUCE-3561] [Umbrella ticket] Performance issues in YARN+MR

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Component/s: mrv2, performance
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 15/Dec/11 01:09
• Updated: 08/May/15 18:11
• Resolved: 08/May/15 18:11

Description

Been working on measuring performance of YARN+MR relative to the 0.20.xx release line together with Karam Singh.

This is an umbrella ticket to track all the issues related to performance.

Sub-Tasks

• Shuffle benchmark is nearly 1.5x slower in 0.23: Closed
• Scan benchmark is more than 1.5x slower in 0.23: Resolved
• AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly: Closed
• Batch jobHistory disk flushes: Closed
• Counters occupy a good part of AM heap: Closed
• Optimize YARN API calls: Resolved
• MR AM slows down due to repeatedly constructing ContainerLaunchContext: Closed
• Extraneous JobConf objects in AM heap: Closed
• Optimize Job's progress calculations in MR AM: Closed
• TaskAttemptListener holds a global lock for all task-updates: Closed
• MR AM's dispatcher is blocked by heartbeats to ResourceManager: Closed
• Compression benchmark run-time increased by 13% in 0.23: Closed
• ContainerLocalizer should request new resources after completing the current one: Closed
• Thread pool for launching containers in MR AM not expanding as expected: Closed
• TaskHeartbeatHandler holds a global lock for all task-updates: Closed
• CapacityScheduler should be more conservative assigning off-switch requests: Closed
• DFSIO read throughput is decreased by 16% in 0.23.1 than Hadoop-0.20.204 on 350 nodes size cluster.: Resolved
• AM-NM RPC calls occasionally takes a long time to respond: Open
• Default AM heartbeat interval should be one second: Closed
• Make gridmix performance on YARN+MR to match or exceed that on 1.0: Resolved
• Tasks may take upto 3 seconds to exit after completion: Closed
• MR AM's ContainerAllocator is assigning the allocated containers very slowly: Closed
• Lower default allocation sizes, fix allocation configurations and document them: Closed
• RackResolver should maintain a cache to avoid repetitive lookups.: Closed
• Data Locality suffers if the AM asks for containers using IPs instead of hostnames: Closed
• Counters are getting calculated twice at job-finish and delaying clients.: Closed
• Counters aggregation slowed down significantly after MAPREDUCE-3749: Closed
• MapReduce job submission time has increased in 0.23 when compared to 0.20.206: Open
• RM scalability runtime is worse than 0.20.204 by 14.2%: Resolved