[SPARK-23130] Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.3, 2.1.0, 2.2.0
• Component/s: SQL
• Labels: bulk-closed thrift

Environment

• Hadoop distributions: HDP 2.5 - 2.6.3.0
• OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4

• Assignee: Unassigned
• Reporter: Sean Roberts
• Created: 17/Jan/18 09:49
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

Spark Thrift is not cleaning up /tmp for files & directories named like:
/tmp/hive/*.pipeout
/tmp/*_resources

There are such a large number that /tmp quickly runs out of inodes causing the partition to be unusable and many services to crash. This is even true when the only jobs submitted are routine service checks.

Used `strace` to show that Spark Thrift is responsible:

strace.out.118864:04:53:49 open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout", O_RDWR|O_CREAT|O_EXCL, 0666) = 134
strace.out.118864:04:53:49 mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0

Those files were left behind, even days later.

---

Example files:

# stat /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
  File: ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fe09h/65033d	Inode: 678         Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    hive)   Gid: ( 1002/  hadoop)
Access: 2017-12-19 04:53:49.126777260 -0600
Modify: 2017-12-19 04:53:49.126777260 -0600
Change: 2017-12-19 04:53:49.126777260 -0600
 Birth: -

# stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
  File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: fe09h/65033d	Inode: 668         Links: 2
Access: (0700/drwx------)  Uid: ( 1000/    hive)   Gid: ( 1002/  hadoop)
Access: 2017-12-19 04:57:38.458937635 -0600
Modify: 2017-12-19 04:53:49.062777216 -0600
Change: 2017-12-19 04:53:49.066777218 -0600
 Birth: -

Showing the large number:

# find /tmp/ -name '*_resources' | wc -l
68340
# find /tmp/hive -name "*.pipeout" | wc -l
51837

Issue Links

• is related to: SPARK-15401 Spark Thrift server creates empty directories in tmp directory on the driver
• is related to: HIVE-6091 Empty pipeout files are created for connection create/close