[AMBARI-9351] AMS crashes after filling 100% disk space in /tmp

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: ambari-metrics
• Epic Link: Ambari Metrics System
• Assignee: Dmytro Sen
• Reporter: Dmytro Sen
• Created: 27/Jan/15 12:11
• Updated: 02/Feb/15 22:54
• Resolved: 27/Jan/15 12:40

Description

/grid/0/var/log/ambari-metrics-collector/hbase-ams-master-1.log contains

2015-01-26 14:19:30,715 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=
/tmp
2015-01-26 14:19:43,053 INFO  [main] server.ZooKeeperServer: Server environment:java.io.tmpd
ir=/tmp

Disk space utilization

[root@-1 yum.repos.d]# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1       10319160 9788980      5996 100% /
tmpfs            7650596       0   7650596   0% /dev/shm
/dev/sdb        51606140 2654028  46330672   6% /grid/0

ls -la /tmp

[root@-1 yum.repos.d]# ls -la /tmp/
total 7060908

drwxrwxrwt.  9 root root      53248 Jan 27 11:29 .
dr-xr-xr-x. 24 root root       4096 Jan 24 14:24 ..
-rw-r--r--.  1 ams  hadoop 12618117 Jan 26 20:54 001f9635-aab6-4341-9178-65a4800027e48906594900780884618.tmp
-rw-r--r--.  1 ams  hadoop 12661373 Jan 26 20:39 00af1ee6-4c86-4bb3-abaf-1ec4e2ee73c17100251832777429788.tmp
-rw-r--r--.  1 ams  hadoop 12653638 Jan 26 19:54 00b3c34b-5b39-46d0-aad3-37053d18ce8e593370814143377630.tmp
-rw-r--r--.  1 ams  hadoop 12649298 Jan 26 20:52 01de551e-6914-481c-b8fe-8a501bdcdb154137668780604129205.tmp
-rw-r--r--.  1 ams  hadoop 12640356 Jan 26 20:54 01ef5692-4a08-40f8-aaaa-a823f43acc4f444122025325606803.tmp
-rw-r--r--.  1 ams  hadoop 12628727 Jan 26 21:01 01fb616c-c1d2-4125-8ffd-82baee29fa718829409805136337687.tmp
-rw-r--r--.  1 ams  hadoop 12627578 Jan 26 19:54 023eff7a-cfd4-420f-8a41-e581b58cde6e5962256337406056608.tmp
-rw-r--r--.  1 ams  hadoop 12643138 Jan 26 20:44 036d6e4d-a034-4ab2-b1a2-4d94a34a8a053777797242598109757.tmp
-rw-r--r--.  1 ams  hadoop 12661735 Jan 26 20:47 03f4ef3e-244d-497b-a019-06a9fff276c88260643562666021800.tmp
-rw-r--r--.  1 ams  hadoop 12644853 Jan 26 20:57 043e0641-d964-4d06-aa1a-a085a896b20d1050013569901896778.tmp

TMP dir should be customized with -Djava.io.tmpdir=/path/to/tmpdir

Issue Links

• links to: Review Board