Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561

[Umbrella ticket] Performance issues in YARN+MR

          Details

          • Type: Bug
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 0.23.0
          • Fix Version/s: None
          • Component/s: mrv2, performance
          • Labels:
            None

            Description

            Been working on measuring performance of YARN+MR relative to the 0.20.xx release line together with Karam Singh.

            This is an umbrella ticket to track all the issues related to performance.

              Attachments

              1.
              		 Shuffle benchmark is nearly 1.5x slower in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
              2.
              		 Scan benchmark is more than 1.5x slower in 0.23 Sub-task Resolved Unassigned
              3.
              		 AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly Sub-task Closed Vinod Kumar Vavilapalli
              4.
              		 Batch jobHistory disk flushes Sub-task Closed Siddharth Seth
              5.
              		 Counters occupy a good part of AM heap Sub-task Closed Vinod Kumar Vavilapalli
              6.
              		 Optimize YARN API calls Sub-task Resolved Vinod Kumar Vavilapalli
              7.
              		 MR AM slows down due to repeatedly constructing ContainerLaunchContext Sub-task Closed Vinod Kumar Vavilapalli
              8.
              		 Extraneous JobConf objects in AM heap Sub-task Closed Vinod Kumar Vavilapalli
              9.
              		 Optimize Job's progress calculations in MR AM Sub-task Closed Vinod Kumar Vavilapalli
              10.
              		 TaskAttemptListener holds a global lock for all task-updates Sub-task Closed Vinod Kumar Vavilapalli
              11.
              		 MR AM's dispatcher is blocked by heartbeats to ResourceManager Sub-task Closed Vinod Kumar Vavilapalli
              12.
              		 Compression benchmark run-time increased by 13% in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
              13.
              		 ContainerLocalizer should request new resources after completing the current one Sub-task Closed Siddharth Seth
              14.
              		 Thread pool for launching containers in MR AM not expanding as expected Sub-task Closed Vinod Kumar Vavilapalli
              15.
              		 TaskHeartbeatHandler holds a global lock for all task-updates Sub-task Closed Siddharth Seth
              16.
              		 CapacityScheduler should be more conservative assigning off-switch requests Sub-task Closed Arun C Murthy
              17.
              		 DFSIO read throughput is decreased by 16% in 0.23.1 than Hadoop-0.20.204 on 350 nodes size cluster. Sub-task Resolved Vinod Kumar Vavilapalli
              18.
              		 AM-NM RPC calls occasionally takes a long time to respond Sub-task Open Unassigned
              19.
              		 Default AM heartbeat interval should be one second Sub-task Closed Hitesh Shah
              20.
              		 Make gridmix performance on YARN+MR to match or exceed that on 1.0 Sub-task Resolved Unassigned
              21.
              		 Tasks may take upto 3 seconds to exit after completion Sub-task Closed Siddharth Seth
              22.
              		 MR AM's ContainerAllocator is assigning the allocated containers very slowly Sub-task Closed Vinod Kumar Vavilapalli
              23.
              		 Lower default allocation sizes, fix allocation configurations and document them Sub-task Closed Harsh J
              24.
              		 RackResolver should maintain a cache to avoid repetitive lookups. Sub-task Closed Vinod Kumar Vavilapalli
              25.
              		 Data Locality suffers if the AM asks for containers using IPs instead of hostnames Sub-task Closed Siddharth Seth
              26.
              		 Counters are getting calculated twice at job-finish and delaying clients. Sub-task Closed Vinod Kumar Vavilapalli
              27.
              		 Counters aggregation slowed down significantly after MAPREDUCE-3749 Sub-task Closed Vinod Kumar Vavilapalli
              28.
              		 MapReduce job submission time has increased in 0.23 when compared to 0.20.206 Sub-task Open Unassigned
              29.
              		 RM scalability runtime is worse than 0.20.204 by 14.2% Sub-task Resolved Unassigned

                Activity

                  People

                  • Assignee:
                    vinodkv Vinod Kumar Vavilapalli
                    Reporter:
                    vinodkv Vinod Kumar Vavilapalli
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    15 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: