Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3561 [Umbrella ticket] Performance issues in YARN+MR
  3. MAPREDUCE-3831

RM scalability runtime is worse than 0.20.204 by 14.2%

        Details

        • Type: Sub-task Sub-task
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 0.23.1
        • Fix Version/s: None
        • Component/s: benchmarks, mrv2, resourcemanager
        • Labels:
          None

          Description

          RM scalability runtime is worse than 0.20.204 by 14.2%

          Overall runtime against 0.20.204 in a 350 nodes cluster is 2155 secs.
          Overall runtime against .23 in 350 nodes cluster is 2462 secs.

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Karam Singh added a comment -

            Settings for For RM scalability comparison :-
            GridMix settings were for both Hadoop-0.20.204 and Hadoop-0.23 :

            gridmix.client.submit.threads=10
gridmix.submit.multiplier=0.01
gridmix.client.pending.queue.depth=100
gridmix.job.type=SLEEPJOB
gridmix.sleep.maptask-only=false
gridmix.sleep.max-map-time=300000
gridmix.sleep.max-reduce-tim=300000
gridmix.sleep.fake-locations=3
gridmix.distributed-cache-emulation.enable=falseemulation.enable>
gridmix.compression-emulation.enable=false

            From JobHistroy Parsing and GridMix client found that :
            Runtime (seconds): 2473
            GridMix Simulation Time Spent: 41mins 8sec
            Workflow End: 2046 (From histroy parsing)

            While looking at GridMix log and JobHistory files :
            1. Found that according to gm client last was completed was 

            	12/02/07 08:32:26 INFO gridmix.JobMonitor: GRIDMIX000029 job_1328600848949_1182) success.
	Whereas the 
	EndTime of Job is : 1328602818684 Tue, 07 Feb 2012 08:20:18

            Which means somehow GridMix got Job completion event 12 minutes after the the actual job got completed .

            2. Similarly acc. to JobHistory last Jo completed was :

            		job_1328600848949_1162: 1328603121882 Tue, 07 Feb 2012 08:25:21
		Whereas according to GridMix client log:
		12/02/07 08:32:08 INFO gridmix.JobMonitor: GRIDMIX000029 (job_1328600848949_1162) success

            Which again means GridMix got job completion event nearly 7 minutes after the actual job got finished.

            Whereas this problem does not exists with Hadoop-0.20.204
            Seems that, Somehow in Hadoop-0.23, GridMix is getting job completion events long after the actual job getting completed

            Show
            Karam Singh added a comment - Settings for For RM scalability comparison :- GridMix settings were for both Hadoop-0.20.204 and Hadoop-0.23 : gridmix.client.submit.threads=10 gridmix.submit.multiplier=0.01 gridmix.client.pending.queue.depth=100 gridmix.job.type=SLEEPJOB gridmix.sleep.maptask-only= false gridmix.sleep.max-map-time=300000 gridmix.sleep.max-reduce-tim=300000 gridmix.sleep.fake-locations=3 gridmix.distributed-cache-emulation.enable=falseemulation.enable> gridmix.compression-emulation.enable= false From JobHistroy Parsing and GridMix client found that : Runtime (seconds): 2473 GridMix Simulation Time Spent: 41mins 8sec Workflow End: 2046 (From histroy parsing) While looking at GridMix log and JobHistory files : 1. Found that according to gm client last was completed was 12/02/07 08:32:26 INFO gridmix.JobMonitor: GRIDMIX000029 job_1328600848949_1182) success. Whereas the EndTime of Job is : 1328602818684 Tue, 07 Feb 2012 08:20:18 Which means somehow GridMix got Job completion event 12 minutes after the the actual job got completed . 2. Similarly acc. to JobHistory last Jo completed was : job_1328600848949_1162: 1328603121882 Tue, 07 Feb 2012 08:25:21 Whereas according to GridMix client log: 12/02/07 08:32:08 INFO gridmix.JobMonitor: GRIDMIX000029 (job_1328600848949_1162) success Which again means GridMix got job completion event nearly 7 minutes after the actual job got finished. Whereas this problem does not exists with Hadoop-0.20.204 Seems that, Somehow in Hadoop-0.23, GridMix is getting job completion events long after the actual job getting completed
            Hide
            Permalink
            Arun C Murthy added a comment -

            This looks like a bug in GridMix. Amar, can you please take a look? Thanks.

            Show
            Arun C Murthy added a comment - This looks like a bug in GridMix. Amar, can you please take a look? Thanks.
            Hide
            Permalink
            Amol Kekre added a comment -

            pls close this jira. RM scalability now matches .204

            Show
            Amol Kekre added a comment - pls close this jira. RM scalability now matches .204
            Hide
            Permalink
            Vinod Kumar Vavilapalli added a comment -

            Thanks for the update Amol.

            Closing this as fixed, couple of the tickets tracked under MAPREDUCE-3561 automatically addressed this I suppose.

            Show
            Vinod Kumar Vavilapalli added a comment - Thanks for the update Amol. Closing this as fixed, couple of the tickets tracked under MAPREDUCE-3561 automatically addressed this I suppose.

              People

              • Assignee:
                Unassigned
                Reporter:
                Vinay Kumar Thota
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development