Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1910

Extra checks in DFS.create() are not necessary.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      DistributedFileSystem.create(path) like this :

        public FSDataOutputStream create(Path f, boolean overwrite,
          int bufferSize, short replication, long blockSize,
          Progressable progress) throws IOException {
          if (exists(f) && !overwrite) {
            throw new IOException("File already exists:"+f);
          }
          Path parent = f.getParent();
          if (parent != null && !exists(parent) && !mkdirs(parent)) {
            throw new IOException("Mkdirs failed to create " + parent);
          }
      
          return new FSDataOutputStream( dfs.create(getPathName(f), overwrite, 
                                                    replication, blockSize, 
                                                    progress, bufferSize, ticket));
        }
      

      This has overhead of 2-3 RPCs to namenode for every create(). The first exists() is not required because overwrite flag is passed to Namenode. The second exists() and mkdirs() is not required since create() already does this.

        Attachments

        1. HADOOP-1910.patch
          0.9 kB
          Raghu Angadi

          Activity

            People

            • Assignee:
              rangadi Raghu Angadi
              Reporter:
              rangadi Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: