[SPARK-24059] When blacklist disable always hash to a bad local directory may cause job failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: zhoukang
• Created: 24/Apr/18 01:55
• Updated: 08/Oct/19 05:44
• Resolved: 08/Oct/19 05:44

Description

When blacklist disable , if we always hashed temp shuffle to a bad local directory on the same executor will cause job failure.
Like below:

java.io.FileNotFoundException: /home/work/hdd8/yarn/xxx/nodemanager/usercache/xxx/appcache/application_1520502842490_20813/blockmgr-3beeddbf-bb83-4a74-ad7a-e796fe592b7c/27/temp_shuffle_159e6886-b76f-4d96-9600-aee62ada0fa9 (Read-only file system)

java.io.FileNotFoundException: /home/work/hdd8/yarn/xxx/nodemanager/usercache/xxx/appcache/application_1520502842490_20813/blockmgr-3beeddbf-bb83-4a74-ad7a-e796fe592b7c/06/temp_shuffle_ba7f0a29-8e02-4ffa-94f7-01f72d214821 (Read-only file system)

java.io.FileNotFoundException: /home/work/hdd8/yarn/xxx/nodemanager/usercache/xxx/appcache/application_1520502842490_20813/blockmgr-3beeddbf-bb83-4a74-ad7a-e796fe592b7c/32/temp_shuffle_7030256c-fc24-4d45-a901-be23c2c3fbd6 (Read-only file system)

java.io.FileNotFoundException: /home/work/hdd8/yarn/zjyprc-hadoop/nodemanager/usercache/h_message_push/appcache/application_1520502842490_20813/blockmgr-3beeddbf-bb83-4a74-ad7a-e796fe592b7c/14/temp_shuffle_65816622-6217-43b9-bc9e-e2f67dc9a9de (Read-only file system)