[SPARK-4323] Utils#fetchFile method should close lock file certainly - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Not A Problem
Affects Version/s: 1.3.0
Fix Version/s: None
Component/s: Spark Core
Labels:
None

Target Version/s:

1.2.0, 1.3.0

Description

In Utils#fetchFile method, lock file is created like as follows.

      val raf = new RandomAccessFile(lockFile, "rw")                                                                                                          
      // Only one executor entry.                                                                                                                             
      // The FileLock is only used to control synchronization for executors download file,                                                                    
      // it's always safe regardless of lock type (mandatory or advisory).                                                                                    
      val lock = raf.getChannel().lock()                                                                                                                      
      val cachedFile = new File(localDir, cachedFileName)                                                                                                     
      try {                                                                                                                                                   
        if (!cachedFile.exists()) {                                                                                                                           
          doFetchFile(url, localDir, cachedFileName, conf, securityMgr, hadoopConf)                                                                           
        }                                                                                                                                                     
      } finally {                                                                                                                                             
        lock.release()                                                                                                                                        
      }

If some error occurs between opening RandomAccessFile and getting lock, lock file can be not closed.