Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561

[Umbrella ticket] Performance issues in YARN+MR

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: mrv2, performance
    • Labels:
      None

      Description

      Been working on measuring performance of YARN+MR relative to the 0.20.xx release line together with Karam Singh.

      This is an umbrella ticket to track all the issues related to performance.

      1.
      Shuffle benchmark is nearly 1.5x slower in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
       
      2.
      Scan benchmark is more than 1.5x slower in 0.23 Sub-task Resolved Unassigned
       
      3.
      AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly Sub-task Closed Vinod Kumar Vavilapalli
       
      4.
      Batch jobHistory disk flushes Sub-task Closed Siddharth Seth
       
      5.
      Counters occupy a good part of AM heap Sub-task Closed Vinod Kumar Vavilapalli
       
      6. Optimize YARN API calls Sub-task Open Vinod Kumar Vavilapalli
       
      7.
      MR AM slows down due to repeatedly constructing ContainerLaunchContext Sub-task Closed Vinod Kumar Vavilapalli
       
      8.
      Extraneous JobConf objects in AM heap Sub-task Closed Vinod Kumar Vavilapalli
       
      9.
      Optimize Job's progress calculations in MR AM Sub-task Closed Vinod Kumar Vavilapalli
       
      10.
      TaskAttemptListener holds a global lock for all task-updates Sub-task Closed Vinod Kumar Vavilapalli
       
      11.
      MR AM's dispatcher is blocked by heartbeats to ResourceManager Sub-task Closed Vinod Kumar Vavilapalli
       
      12.
      Compression benchmark run-time increased by 13% in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
       
      13.
      ContainerLocalizer should request new resources after completing the current one Sub-task Closed Siddharth Seth
       
      14.
      Thread pool for launching containers in MR AM not expanding as expected Sub-task Closed Vinod Kumar Vavilapalli
       
      15.
      TaskHeartbeatHandler holds a global lock for all task-updates Sub-task Closed Siddharth Seth
       
      16.
      CapacityScheduler should be more conservative assigning off-switch requests Sub-task Closed Arun C Murthy
       
      17.
      DFSIO read throughput is decreased by 16% in 0.23.1 than Hadoop-0.20.204 on 350 nodes size cluster. Sub-task Resolved Vinod Kumar Vavilapalli
       
      18. AM-NM RPC calls occasionally takes a long time to respond Sub-task Open Unassigned
       
      19.
      Default AM heartbeat interval should be one second Sub-task Closed Hitesh Shah
       
      20.
      Make gridmix performance on YARN+MR to match or exceed that on 1.0 Sub-task Resolved Unassigned
       
      21.
      Tasks may take upto 3 seconds to exit after completion Sub-task Closed Siddharth Seth
       
      22.
      MR AM's ContainerAllocator is assigning the allocated containers very slowly Sub-task Closed Vinod Kumar Vavilapalli
       
      23.
      Lower default allocation sizes, fix allocation configurations and document them Sub-task Closed Harsh J
       
      24.
      RackResolver should maintain a cache to avoid repetitive lookups. Sub-task Closed Vinod Kumar Vavilapalli
       
      25.
      Data Locality suffers if the AM asks for containers using IPs instead of hostnames Sub-task Closed Siddharth Seth
       
      26.
      Counters are getting calculated twice at job-finish and delaying clients. Sub-task Closed Vinod Kumar Vavilapalli
       
      27.
      Counters aggregation slowed down significantly after MAPREDUCE-3749 Sub-task Closed Vinod Kumar Vavilapalli
       
      28. MapReduce job submission time has increased in 0.23 when compared to 0.20.206 Sub-task Open Unassigned
       
      29.
      RM scalability runtime is worse than 0.20.204 by 14.2% Sub-task Resolved Unassigned
       

        Activity

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:

              Development