[SPARK-9827] Too many open files in TungstenExchange

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 11/Aug/15 16:54
• Updated: 13/Aug/15 17:19
• Resolved: 13/Aug/15 03:03

Description

When run q19 on TPCDS (scale=5) dataset with 8G memory, it open 10k shuffle files, crash many things (even Chrome).

davies@localhost:~/work/spark$ jps
95385 Jps
95316 SparkSubmit
davies@localhost:~/work/spark$ lsof -p 95316 | wc -l
9827
davies@localhost:~/work/spark$ lsof -p 95316 | tail
java    95316 davies 9772r     REG                1,2      9522 97350739 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/2a/shuffle_0_112_0.data
java    95316 davies 9773r     REG                1,2      8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
java    95316 davies 9774r     REG                1,2      8200 97351134 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/09/shuffle_0_113_0.data
java    95316 davies 9775r     REG                1,2      8057 97351941 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/05/shuffle_0_117_0.data
java    95316 davies 9776r     REG                1,2      8565 97351133 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/18/shuffle_0_114_0.data
java    95316 davies 9777r     REG                1,2      8185 97351942 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1c/shuffle_0_118_0.data
java    95316 davies 9778r     REG                1,2      8865 97351135 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/07/shuffle_0_115_0.data
java    95316 davies 9779r     REG                1,2      8255 97351987 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/3d/shuffle_0_119_0.data
java    95316 davies 9780r     REG                1,2      8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
java    95316 davies 9781r     REG                1,2      9105 97352148 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/13/shuffle_0_120_0.data

davies@localhost:~/work/spark$ ls -l /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-71afa3af-f2a5-4b72-8b2d-45aa70ff7466//3a/
total 68
-rw-r--r-- 1 davies staff 8272 Aug 11 09:57 shuffle_0_105_0.data
-rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_109_0.index
-rw-r--r-- 1 davies staff 8414 Aug 11 09:57 shuffle_0_127_0.data
-rw-r--r-- 1 davies staff 8368 Aug 11 09:57 shuffle_0_149_0.data
-rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_40_0.index
-rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_62_0.index
-rw-r--r-- 1 davies staff 7965 Aug 11 09:57 shuffle_0_6_0.data
-rw-r--r-- 1 davies staff 8419 Aug 11 09:57 shuffle_0_80_0.data

Issue Links

• duplicates: SPARK-9499 Possible file handle leak in spilling/sort code
• duplicates: SPARK-9921 Too many open files in Spark SQL
• links to: [Github] Pull Request #8116 (davies)