[HDFS-4309] Multithreaded get through the Cache FileSystem Object to lead LeaseChecker memory leak

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.205.0, 0.23.1, 0.23.4, 2.0.1-alpha
• Component/s: hdfs-client
• Labels: BB2015-05-TBR patch
• Target Version/s: 2.0.2-alpha
• Assignee: Unassigned
• Reporter: WenJin Ma
• Created: 13/Dec/12 08:47
• Updated: 06/May/15 03:34

Description

If multiple threads concurrently execute the following methods will result in the thread fs = createFileSystem (uri, conf) method is called.And create multiple DFSClient, start at the same time LeaseChecker daemon thread, may not be able to use shutdownhook close it after the process, resulting in a memory leak.

private FileSystem getInternal(URI uri, Configuration conf, Key key) throws IOException{
      FileSystem fs = null;
      synchronized (this) {
        fs = map.get(key);
      }
      if (fs != null) {
        return fs;
      }
      //  this is 
      fs = createFileSystem(uri, conf);
      synchronized (this) {  // refetch the lock again
        FileSystem oldfs = map.get(key);
        if (oldfs != null) { // a file system is created while lock is releasing
          fs.close(); // close the new file system
          return oldfs;  // return the old file system
        }

        // now insert the new file system into the map
        if (map.isEmpty() && !clientFinalizer.isAlive()) {
          Runtime.getRuntime().addShutdownHook(clientFinalizer);
        }
        fs.key = key;
        map.put(key, fs);

        if (conf.getBoolean("fs.automatic.close", true)) {
          toAutoClose.add(key);
        }
        return fs;
      }
    }