[SPARK-31591] namePrefix could be null in Utils.createDirectory

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Lantao Jin
• Created: 28/Apr/20 06:07
• Updated: 29/Apr/20 04:05
• Resolved: 29/Apr/20 02:08

Description

In our production, we find that many shuffle files could be located in
/hadoop/2/yarn/local/usercache/b_carmel/appcache/application_1586487864336_4602/null-107d4e9c-d3c7-419e-9743-a21dc4eaeb3f/3a

The Util.createDirectory() uses a default parameter "spark"

  def createDirectory(root: String, namePrefix: String = "spark"): File = {

But in some cases, the actual namePrefix is null. If the method is called with null, then the default value would not be applied.