Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1910

Extra checks in DFS.create() are not necessary.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.14.1
    • 0.15.0
    • None
    • None

    Description

      DistributedFileSystem.create(path) like this :

        public FSDataOutputStream create(Path f, boolean overwrite,
          int bufferSize, short replication, long blockSize,
          Progressable progress) throws IOException {
          if (exists(f) && !overwrite) {
            throw new IOException("File already exists:"+f);
          }
          Path parent = f.getParent();
          if (parent != null && !exists(parent) && !mkdirs(parent)) {
            throw new IOException("Mkdirs failed to create " + parent);
          }
      
          return new FSDataOutputStream( dfs.create(getPathName(f), overwrite, 
                                                    replication, blockSize, 
                                                    progress, bufferSize, ticket));
        }
      

      This has overhead of 2-3 RPCs to namenode for every create(). The first exists() is not required because overwrite flag is passed to Namenode. The second exists() and mkdirs() is not required since create() already does this.

      Attachments

        1. HADOOP-1910.patch
          0.9 kB
          Raghu Angadi

        Activity

          People

            rangadi Raghu Angadi
            rangadi Raghu Angadi
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: