Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.2
-
None
-
OS: Ubuntu 18.04
JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB ssd.
37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB ssd.
2 * 3.5 Gb slots per Node Manager, 1x cpu per slotyarn-site: yarn-site.xml
yarn-client-yarn-site: yarn-client.yarn-site.xmlOS: Ubuntu 18.04 JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB ssd. 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB ssd. 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot yarn-site: yarn-site.xml yarn-client-yarn-site: yarn-client.yarn-site.xml
Description
Opportunistic scheduling is supposed to provide lower scheduling time, and thus higher task throughput and lower job completion times for short jobs/tasks.
Through my experiments I have found distributed scheduling can degrade performance.
I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run time was 1523ms.
Findings:
- Job completion time, the time take from submitting a job to job completion, may degrade by over 200%
jct_cdf_100j_100t_1500.svg
jct_cdf_100j_50t_1500_with_outliers.svg - Job execution time may increase by up to 300%
jet_boxplot_j100_50t_1500.svg
jet_boxplot_j100_50t_1500_with_outliers.svg - Task throughput decreased by 100%
task_throughput_boxplot_100j_50t_1500.svg