Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561

[Umbrella ticket] Performance issues in YARN+MR

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: mrv2, performance
    • Labels:
      None

      Description

      Been working on measuring performance of YARN+MR relative to the 0.20.xx release line together with Karam Singh.

      This is an umbrella ticket to track all the issues related to performance.

        Attachments

        1.
        Shuffle benchmark is nearly 1.5x slower in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
        2.
        Scan benchmark is more than 1.5x slower in 0.23 Sub-task Resolved Unassigned
        3.
        AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly Sub-task Closed Vinod Kumar Vavilapalli
        4.
        Batch jobHistory disk flushes Sub-task Closed Siddharth Seth
        5.
        Counters occupy a good part of AM heap Sub-task Closed Vinod Kumar Vavilapalli
        6.
        Optimize YARN API calls Sub-task Resolved Vinod Kumar Vavilapalli
        7.
        MR AM slows down due to repeatedly constructing ContainerLaunchContext Sub-task Closed Vinod Kumar Vavilapalli
        8.
        Extraneous JobConf objects in AM heap Sub-task Closed Vinod Kumar Vavilapalli
        9.
        Optimize Job's progress calculations in MR AM Sub-task Closed Vinod Kumar Vavilapalli
        10.
        TaskAttemptListener holds a global lock for all task-updates Sub-task Closed Vinod Kumar Vavilapalli
        11.
        MR AM's dispatcher is blocked by heartbeats to ResourceManager Sub-task Closed Vinod Kumar Vavilapalli
        12.
        Compression benchmark run-time increased by 13% in 0.23 Sub-task Closed Vinod Kumar Vavilapalli
        13.
        ContainerLocalizer should request new resources after completing the current one Sub-task Closed Siddharth Seth
        14.
        Thread pool for launching containers in MR AM not expanding as expected Sub-task Closed Vinod Kumar Vavilapalli
        15.
        TaskHeartbeatHandler holds a global lock for all task-updates Sub-task Closed Siddharth Seth
        16.
        CapacityScheduler should be more conservative assigning off-switch requests Sub-task Closed Arun C Murthy
        17.
        DFSIO read throughput is decreased by 16% in 0.23.1 than Hadoop-0.20.204 on 350 nodes size cluster. Sub-task Resolved Vinod Kumar Vavilapalli
        18.
        AM-NM RPC calls occasionally takes a long time to respond Sub-task Open Unassigned
        19.
        Default AM heartbeat interval should be one second Sub-task Closed Hitesh Shah
        20.
        Make gridmix performance on YARN+MR to match or exceed that on 1.0 Sub-task Resolved Unassigned
        21.
        Tasks may take upto 3 seconds to exit after completion Sub-task Closed Siddharth Seth
        22.
        MR AM's ContainerAllocator is assigning the allocated containers very slowly Sub-task Closed Vinod Kumar Vavilapalli
        23.
        Lower default allocation sizes, fix allocation configurations and document them Sub-task Closed Harsh J
        24.
        RackResolver should maintain a cache to avoid repetitive lookups. Sub-task Closed Vinod Kumar Vavilapalli
        25.
        Data Locality suffers if the AM asks for containers using IPs instead of hostnames Sub-task Closed Siddharth Seth
        26.
        Counters are getting calculated twice at job-finish and delaying clients. Sub-task Closed Vinod Kumar Vavilapalli
        27.
        Counters aggregation slowed down significantly after MAPREDUCE-3749 Sub-task Closed Vinod Kumar Vavilapalli
        28.
        MapReduce job submission time has increased in 0.23 when compared to 0.20.206 Sub-task Open Unassigned
        29.
        RM scalability runtime is worse than 0.20.204 by 14.2% Sub-task Resolved Unassigned

          Activity

            People

            • Assignee:
              vinodkv Vinod Kumar Vavilapalli
              Reporter:
              vinodkv Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: