[SPARK-4323] Utils#fetchFile method should close lock file certainly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.2.0, 1.3.0
• Assignee: Unassigned
• Reporter: Kousuke Saruta
• Created: 10/Nov/14 19:32
• Updated: 16/Dec/14 00:28
• Resolved: 16/Dec/14 00:28

Description

In Utils#fetchFile method, lock file is created like as follows.

      val raf = new RandomAccessFile(lockFile, "rw")                                                                                                          
      // Only one executor entry.                                                                                                                             
      // The FileLock is only used to control synchronization for executors download file,                                                                    
      // it's always safe regardless of lock type (mandatory or advisory).                                                                                    
      val lock = raf.getChannel().lock()                                                                                                                      
      val cachedFile = new File(localDir, cachedFileName)                                                                                                     
      try {                                                                                                                                                   
        if (!cachedFile.exists()) {                                                                                                                           
          doFetchFile(url, localDir, cachedFileName, conf, securityMgr, hadoopConf)                                                                           
        }                                                                                                                                                     
      } finally {                                                                                                                                             
        lock.release()                                                                                                                                        
      }  

If some error occurs between opening RandomAccessFile and getting lock, lock file can be not closed.

Issue Links

• links to: [Github] Pull Request #3186 (sarutak)