Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15515

adl.AdlFilesystem.close() doesn't release locks on open files



    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.7.3
    • None
    • fs/adl
    • None


      If you write to a file on and Azure ADL filesystem and close the file system but not the file before the process exits, the next time you try open the file for append it fails with:

      Exception in thread "main" java.io.IOException: APPEND failed with error 0x83090a16 (Failed to perform the requested operation because the file is currently open in write mode by another user or process.). [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]

       The following moves local file to HDFS if it doesn't exist or appends it's contents if it does:


      public void addFile(String source, String dest, Configuration conf) throws IOException {
      FileSystem fileSystem = FileSystem.get(conf);
      // Get the filename out of the file path
      String filename = source.substring(source.lastIndexOf('/') + 1,source.length());
      // Create the destination path including the filename.
      if (dest.charAt(dest.length() - 1) != '/')
      { dest = dest + "/" + filename; }
      else {
      dest = dest + filename;
      // Check if the file already exists
      Path path = new Path(dest);
      FSDataOutputStream out;
      if (fileSystem.exists(path)) {
      System.out.println("File " + dest + " already exists appending");
      out = fileSystem.append(path);
      } else {
      out = fileSystem.create(path);
      // Create a new file and write data to it.
      InputStream in = new BufferedInputStream(new FileInputStream(new File(
      byte[] b = new byte[1024];
      int numBytes = 0;
      while ((numBytes = in.read(b)) > 0) {
      out.write(b, 0, numBytes);
      // Close the file system not the file

      If "dest" is an adl:// location, invoking the function a second time (after the process has exited) it raises the error. If it's a regular hdfs:// file system, it doesn't as all the locks are released. The same exception is also raised if a subsequent append is done using: hdfs dfs  -appendToFile.

      As I can't see a way to force lease recovery in this situation, this seems like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from org.apache.hadoop.fs.FileSystem


      Which states:

      Close this FileSystem instance. Will release any held locks. This does not seem to be the case


        1. HADOOP-15515-001.patch
          18 kB
          Vishwajeet Dusane



            vishwajeet.dusane Vishwajeet Dusane
            ukjay75 Jay Hankinson
            0 Vote for this issue
            4 Start watching this issue