Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5438

Merge FileSystem.create and FileSystem.append

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • fs
    • None
    • Incompatible change

    Description

      Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
      the code looks like:

      FSDataOutputStream out_1 = null;
      if (fs.exists(path_1))
         out_1 = fs.append(path_1);
      else
         out_1 = fs.create(path_1);
      

      . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.

      It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:

      boolean exists = fs.exists(name);
      if (overwrite) {
          if (exists)
             fs.delete(name, true);
           this.out = fs.create(name, overwrite, bufferSize, replication,
      				    blockSize, progress);
           this.currentRowID = 0;
       } else {
         if (!exists)
      	this.out = fs.create(name, overwrite, bufferSize,
      					replication, blockSize, progress);
         else
      	this.out = fs.append(name, bufferSize, progress);
      

      Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.

      BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

      Attachments

        1. Hadoop-5438-2009-03-30.patch
          31 kB
          He Yongqiang
        2. Hadoop-5438-2009-03-31.patch
          34 kB
          He Yongqiang
        3. Hadoop-5438-2009-03-31-2.patch
          34 kB
          He Yongqiang
        4. Hadoop-5438(2009-04-06).patch
          39 kB
          He Yongqiang
        5. Hadoop-5438-2009-05-5.patch
          39 kB
          He Yongqiang
        6. Hadoop-5438-2009-05-10.patch
          39 kB
          He Yongqiang
        7. Hadoop-5438-2009-05-15.patch
          34 kB
          He Yongqiang
        8. Hadoop-5438-2009-05-19.patch
          34 kB
          He Yongqiang

        Issue Links

          Activity

            People

              he yongqiang He Yongqiang
              he yongqiang He Yongqiang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: