[AMBARI-9771] AMS: ResultSpooler spill files are left behind in /tmp folder

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: ambari-metrics
• Epic Link: Ambari Metrics System
• Assignee: Dmytro Sen
• Reporter: Dmytro Sen
• Created: 24/Feb/15 15:13
• Updated: 25/Feb/15 13:00
• Resolved: 25/Feb/15 11:42

Description

I've seen behavior, described in PHOENIX-1395, on Ambari cluster (200 nodes)

After 3 days running, /tmp folder is full of ResultSpool files

[root@perf-200-20feb-b-ams-1 ~]# ls -la /tmp/
total 4238768
drwxrwxrwt.  6 root root        4096 Feb 23 10:13 .
dr-xr-xr-x. 24 root root        4096 Feb 20 14:55 ..
drwxr-xr-x.  2 hdfs root        4096 Feb 20 16:46 hadoop-hdfs
drwxr-xr-x.  2 ams  hadoop      4096 Feb 20 18:19 hsperfdata_ams
drwxrwxrwt.  2 root root        4096 Feb 20 14:16 .ICE-unix
drwxr-xr-x.  5 ams  hadoop      4096 Feb 20 18:20 Jetty_0_0_0_0_6188_applicationhistory____g7149g
-rw-r--r--.  1 root root         617 Feb 20 14:16 keyfile.1115.pub
-rwxrwxrwx.  1 root root     2557032 Feb 20 17:59 libyjpagent.so
-rw-r--r--.  1 ams  hadoop  16676205 Feb 22 11:36 ResultSpooler1083719173569401088.bin
-rw-r--r--.  1 ams  hadoop  71227095 Feb 22 02:10 ResultSpooler1095642585411691664.bin
-rw-r--r--.  1 ams  hadoop  48685977 Feb 22 11:32 ResultSpooler1119370459591752039.bin
-rw-r--r--.  1 ams  hadoop  64366788 Feb 21 02:24 ResultSpooler1249566735083013570.bin
-rw-r--r--.  1 ams  hadoop 102519737 Feb 21 15:15 ResultSpooler1434399174629975447.bin
-rw-r--r--.  1 ams  hadoop  62280357 Feb 21 07:04 ResultSpooler1523399749056056116.bin
-rw-r--r--.  1 ams  hadoop  99096145 Feb 21 23:28 ResultSpooler1591731371355221216.bin
-rw-r--r--.  1 ams  hadoop  33274946 Feb 22 07:42 ResultSpooler1763163145889847809.bin
-rw-r--r--.  1 ams  hadoop  29906801 Feb 22 06:45 ResultSpooler1924668033838582502.bin
-rw-r--r--.  1 ams  hadoop  79817760 Feb 20 18:19 ResultSpooler2061194340898298157.bin
-rw-r--r--.  1 ams  hadoop  39832796 Feb 22 11:32 ResultSpooler2084948115978783352.bin
-rw-r--r--.  1 ams  hadoop  16340565 Feb 22 11:07 ResultSpooler2151011123128612740.bin
-rw-r--r--.  1 ams  hadoop  71125739 Feb 21 12:44 ResultSpooler227400912270327467.bin
-rw-r--r--.  1 ams  hadoop  47016101 Feb 21 15:15 ResultSpooler2397265359677018278.bin
-rw-r--r--.  1 ams  hadoop  96264436 Feb 22 02:10 ResultSpooler2614183649855534427.bin
-rw-r--r--.  1 ams  hadoop  69682164 Feb 20 18:19 ResultSpooler2614354335568461108.bin
-rw-r--r--.  1 ams  hadoop  16215764 Feb 22 06:45 ResultSpooler3246758863501341303.bin
-rw-r--r--.  1 ams  hadoop 148128967 Feb 20 20:50 ResultSpooler3488123938708977657.bin
-rw-r--r--.  1 ams  hadoop  25723333 Feb 22 06:46 ResultSpooler3579970151989744696.bin
-rw-r--r--.  1 ams  hadoop  21823041 Feb 22 07:42 ResultSpooler3777629950823001041.bin
-rw-r--r--.  1 ams  hadoop  72062056 Feb 20 18:04 ResultSpooler3852071445687640038.bin
-rw-r--r--.  1 ams  hadoop  81368534 Feb 21 01:22 ResultSpooler3875643246636160232.bin
-rw-r--r--.  1 ams  hadoop  30168849 Feb 22 10:25 ResultSpooler3965157245482339436.bin
-rw-r--r--.  1 ams  hadoop 196130729 Feb 20 18:08 ResultSpooler4121123109947014705.bin
-rw-r--r--.  1 ams  hadoop  23298601 Feb 22 06:45 ResultSpooler4193988685099849898.bin
-rw-r--r--.  1 ams  hadoop  24327567 Feb 22 11:08 ResultSpooler435113448003575788.bin
-rw-r--r--.  1 ams  hadoop  98370825 Feb 21 20:57 ResultSpooler4731594258931869098.bin
-rw-r--r--.  1 ams  hadoop  81147072 Feb 22 05:17 ResultSpooler4839918557747068904.bin
-rw-r--r--.  1 ams  hadoop  16321500 Feb 22 11:40 ResultSpooler4948059449385475969.bin
-rw-r--r--.  1 ams  hadoop  26490333 Feb 22 11:32 ResultSpooler5183367322708173382.bin
-rw-r--r--.  1 ams  hadoop  41887967 Feb 22 02:09 ResultSpooler5363100158650913466.bin
-rw-r--r--.  1 ams  hadoop  61564579 Feb 22 03:22 ResultSpooler5447269850518103716.bin
-rw-r--r--.  1 ams  hadoop 102368545 Feb 21 05:23 ResultSpooler5708767266121623071.bin
-rw-r--r--.  1 ams  hadoop  27115177 Feb 21 12:24 ResultSpooler5742673986917724242.bin
-rw-r--r--.  1 ams  hadoop 136561367 Feb 22 02:48 ResultSpooler5764784857800306062.bin
-rw-r--r--.  1 ams  hadoop  72956934 Feb 21 00:05 ResultSpooler5802156014065836906.bin
-rw-r--r--.  1 ams  hadoop 158032579 Feb 20 20:20 ResultSpooler5957862203905947616.bin
-rw-r--r--.  1 ams  hadoop 163713067 Feb 21 15:17 ResultSpooler6028348699797622192.bin
-rw-r--r--.  1 ams  hadoop  74337617 Feb 21 15:17 ResultSpooler6513985637309952930.bin
-rw-r--r--.  1 ams  hadoop 151691821 Feb 22 05:17 ResultSpooler6722852862052886883.bin
-rw-r--r--.  1 ams  hadoop 158153349 Feb 22 02:12 ResultSpooler6785318486215835763.bin
-rw-r--r--.  1 ams  hadoop  63085961 Feb 21 22:22 ResultSpooler7037247582202398147.bin
-rw-r--r--.  1 ams  hadoop  25750043 Feb 22 11:07 ResultSpooler7161010036991333211.bin
-rw-r--r--.  1 ams  hadoop  23637237 Feb 22 11:32 ResultSpooler7265305658142312639.bin
-rw-r--r--.  1 ams  hadoop  23514267 Feb 22 10:25 ResultSpooler7273301530051464668.bin
-rw-r--r--.  1 ams  hadoop  23725265 Feb 22 07:41 ResultSpooler7435219196600981746.bin
-rw-r--r--.  1 ams  hadoop  74368269 Feb 21 07:04 ResultSpooler7530898046096394659.bin
-rw-r--r--.  1 ams  hadoop  20624014 Feb 20 18:00 ResultSpooler7621079742935581555.bin
-rw-r--r--.  1 ams  hadoop  16321827 Feb 22 07:41 ResultSpooler7731471884268988990.bin
-rw-r--r--.  1 ams  hadoop 134992041 Feb 21 23:28 ResultSpooler7749761859495380220.bin
-rw-r--r--.  1 ams  hadoop  16246582 Feb 22 11:32 ResultSpooler7883369468940765812.bin
-rw-r--r--.  1 ams  hadoop  38676810 Feb 22 11:07 ResultSpooler8060777733113377059.bin
-rw-r--r--.  1 ams  hadoop  66444927 Feb 21 20:22 ResultSpooler8171687820667276916.bin
-rw-r--r--.  1 ams  hadoop 101361125 Feb 21 22:00 ResultSpooler8235372529929072797.bin
-rw-r--r--.  1 ams  hadoop  98019115 Feb 20 21:47 ResultSpooler8258396922291011919.bin
-rw-r--r--.  1 ams  hadoop  98958044 Feb 21 14:34 ResultSpooler841310175906717024.bin
-rw-r--r--.  1 ams  hadoop  37453544 Feb 21 07:02 ResultSpooler8476786944095537854.bin
-rw-r--r--.  1 ams  hadoop  40917633 Feb 22 10:26 ResultSpooler86283493320653508.bin
-rw-r--r--.  1 ams  hadoop  16339554 Feb 22 10:25 ResultSpooler8650910217102203243.bin
-rw-r--r--.  1 ams  hadoop  33496406 Feb 21 23:03 ResultSpooler8840476939367711195.bin
-rw-r--r--.  1 ams  hadoop  34168131 Feb 22 10:26 ResultSpooler8866025986224263370.bin
-rw-r--r--.  1 ams  hadoop 145475909 Feb 20 23:19 ResultSpooler908545909889857121.bin
-rw-r--r--.  1 ams  hadoop 145899169 Feb 20 21:49 ResultSpooler9088480868111255890.bin
-rw-r--r--.  1 ams  hadoop 100219001 Feb 20 20:19 ResultSpooler9184998486224339551.bin

Issue Links

• links to: Review Board