Hadoop Common
  1. Hadoop Common
  2. HADOOP-1708

make files visible in the namespace as soon as they are created

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      In the current DFS implementation, a file appears in the namespace only when the creator closes the file. Also, if the namenode or the client dies before closing the file, the file never appears in the namespace.

      This issue will make files appear in the namespace as soon as it is created. Also, it will continue to remain in the namespace even if the creator dies before closing the file.

      This is related to HADOOP-89. It is different from HADOOP-89 because it does not attempt to make data visible as soon as it is written.

      1. 1708-20070820a.patch
        2 kB
        Christophe Taton
      2. atomicCreation2.patch
        8 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          Updates patch merged with latest trunk.

          Show
          dhruba borthakur added a comment - Updates patch merged with latest trunk.
          Hide
          Hairong Kuang added a comment -

          +1 The code looks good.

          Show
          Hairong Kuang added a comment - +1 The code looks good.
          Hide
          dhruba borthakur added a comment -

          I just committed this.

          Show
          dhruba borthakur added a comment - I just committed this.
          Hide
          Christophe Taton added a comment -

          Is it the right behavior, to keep the file directory entry even if the creator calls abandonFileInProgress?

          Show
          Christophe Taton added a comment - Is it the right behavior, to keep the file directory entry even if the creator calls abandonFileInProgress?
          Hide
          Christophe Taton added a comment -

          I would suggest this modification which removes the file entry from the directory tree when a client abandons a file being created.

          • added a test case for this

          JUnit tests pass successfully.

          Show
          Christophe Taton added a comment - I would suggest this modification which removes the file entry from the directory tree when a client abandons a file being created. added a test case for this JUnit tests pass successfully.
          Hide
          Sameer Paranjpye added a comment -

          -1

          The presence of abandonFileInProgress was a necessary side effect of files not being visible when they were created. Now that they are, this is no longer needed. In any case, the only reliable way to prevent clients from stepping on each others toes is to use temporary filenames and then promote them to final names as is done in map/reduce. A client that needs to call abandonFileInProgress as your test case does can just as easily call delete.

          Show
          Sameer Paranjpye added a comment - -1 The presence of abandonFileInProgress was a necessary side effect of files not being visible when they were created. Now that they are, this is no longer needed. In any case, the only reliable way to prevent clients from stepping on each others toes is to use temporary filenames and then promote them to final names as is done in map/reduce. A client that needs to call abandonFileInProgress as your test case does can just as easily call delete.
          Hide
          Christophe Taton added a comment -

          Should we merge abandonFileInProgress into delete then, and mark abandonFileInProgress as deprecated?

          Show
          Christophe Taton added a comment - Should we merge abandonFileInProgress into delete then, and mark abandonFileInProgress as deprecated?
          Hide
          Sameer Paranjpye added a comment -

          I think that makes sense.

          Show
          Sameer Paranjpye added a comment - I think that makes sense.
          Hide
          Christophe Taton added a comment -

          Anyway, a pending create which is never completed (e.g. because the client died) should remove the file entry... As well, to preserve some compatibility with the current semantic, the abandonFileInProgress operation need to be updated to delete the file entry.

          Show
          Christophe Taton added a comment - Anyway, a pending create which is never completed (e.g. because the client died) should remove the file entry... As well, to preserve some compatibility with the current semantic, the abandonFileInProgress operation need to be updated to delete the file entry.
          Hide
          Raghu Angadi added a comment -

          Before this patch, completeInternal() looked up file INode twice. Since a fileINode is supposed to exist with this patch, we can avoid one INode look up. INode lookup is a little costly since it parses the full path and iterates down the directory tree.

          Show
          Raghu Angadi added a comment - Before this patch, completeInternal() looked up file INode twice. Since a fileINode is supposed to exist with this patch, we can avoid one INode look up. INode lookup is a little costly since it parses the full path and iterates down the directory tree.
          Hide
          Raghu Angadi added a comment -

          What should the semantics of other file modications like reaname(), delete() be? Currently it looks like file can be created and before it is closed some one could rename it. Of course close() will fail.

          Show
          Raghu Angadi added a comment - What should the semantics of other file modications like reaname(), delete() be? Currently it looks like file can be created and before it is closed some one could rename it. Of course close() will fail.
          Hide
          Raghu Angadi added a comment -

          I think we should revert this patch until the semantics are are clear. See HADOOP-1765 for one of the affected cases.

          Show
          Raghu Angadi added a comment - I think we should revert this patch until the semantics are are clear. See HADOOP-1765 for one of the affected cases.
          Hide
          Christophe Taton added a comment -

          I agree. At least in a short term, the semantics should remain as it is when the file is not visible, meaning that a file being created cannot be deleted or renamed before it is completed.
          Ideally however, renaming and deletion should also work on file being created. Renaming should not interfere with the file creation, while deletion should prevent it to complete.

          Show
          Christophe Taton added a comment - I agree. At least in a short term, the semantics should remain as it is when the file is not visible, meaning that a file being created cannot be deleted or renamed before it is completed. Ideally however, renaming and deletion should also work on file being created. Renaming should not interfere with the file creation, while deletion should prevent it to complete.
          Hide
          Sameer Paranjpye added a comment -

          > Anyway, a pending create which is never completed ...

          Why? Yes, it is a departure from the current semantics, but something that is quite intentional. Calling delete should be the responsibility of user code not the client library. This issue is one of the steps towards introducing append. With appends in place, we want exactly the behavior that when a client dies we don't delete a pending file, so that it or another client can resume where it left off.

          > Ideally however, renaming and deletion should also work on file being created.

          So, rather than revert the change, I'd vote for implementing this behavior.

          Show
          Sameer Paranjpye added a comment - > Anyway, a pending create which is never completed ... Why? Yes, it is a departure from the current semantics, but something that is quite intentional. Calling delete should be the responsibility of user code not the client library. This issue is one of the steps towards introducing append. With appends in place, we want exactly the behavior that when a client dies we don't delete a pending file, so that it or another client can resume where it left off. > Ideally however, renaming and deletion should also work on file being created. So, rather than revert the change, I'd vote for implementing this behavior.
          Hide
          Konstantin Shvachko added a comment -

          Yes we knew this changes the semantics of file creates, but we did not take into account that semantics of deletes/renames will change too.
          I do not think we have a consensus on what it should be.
          Imho until we define the semantics and implement it the patch should be reverted (if it is still possible).

          > Ideally however, renaming and deletion should also work on file being created.

          Would it be more consistent to forbid deletion of files that have active leases.
          I agree rename should work on being created file, but then we will have to introduce file ids for accessing files that have been opened.

          Show
          Konstantin Shvachko added a comment - Yes we knew this changes the semantics of file creates, but we did not take into account that semantics of deletes/renames will change too. I do not think we have a consensus on what it should be. Imho until we define the semantics and implement it the patch should be reverted (if it is still possible). > Ideally however, renaming and deletion should also work on file being created. Would it be more consistent to forbid deletion of files that have active leases. I agree rename should work on being created file, but then we will have to introduce file ids for accessing files that have been opened.
          Hide
          Sameer Paranjpye added a comment -

          > Yes we knew this changes the semantics of file creates, but we did not take into account that semantics of deletes/renames will change too.

          How not? The semantics for delete and rename for files that are being written are consistent with the semantics of delete and rename for files that are in the namespace and that have been completed and closed. If a client is reading a 'complete' file and that file gets deleted or renamed then the reader will fail. Arguably, this is a flaw in the filesystem, but addressing that is a different discussion.

          All this issue does is make files visible in the namespace as they are created and while they are being written. Delete and rename semantics for these files are the same as for all other files.

          > Would it be more consistent to forbid deletion of files that have active leases.

          It might be, but the change should apply to all files that are being written or read. I guess that is what you mean by files with active leases. It might be better to fix it so that files can be deleted and renamed as they are being operated on, with POSIX semantics. We should file a different JIRA issue for this.

          Show
          Sameer Paranjpye added a comment - > Yes we knew this changes the semantics of file creates, but we did not take into account that semantics of deletes/renames will change too. How not? The semantics for delete and rename for files that are being written are consistent with the semantics of delete and rename for files that are in the namespace and that have been completed and closed. If a client is reading a 'complete' file and that file gets deleted or renamed then the reader will fail. Arguably, this is a flaw in the filesystem, but addressing that is a different discussion. All this issue does is make files visible in the namespace as they are created and while they are being written. Delete and rename semantics for these files are the same as for all other files. > Would it be more consistent to forbid deletion of files that have active leases. It might be, but the change should apply to all files that are being written or read. I guess that is what you mean by files with active leases. It might be better to fix it so that files can be deleted and renamed as they are being operated on, with POSIX semantics. We should file a different JIRA issue for this.
          Hide
          Raghu Angadi added a comment -

          There is another more immediate issue with the patch. If a file is started but is not closed, when NameNode is restarted, it appears as a directory while loading the fsimage. Currently if number of blocks is zero, then it is treated as a directory in while loading fsimage. Should we file another jira for it?

          I was working with patch for HADOOP-1298 which has directory specific fields, I saw for the first time an EOF error while loading the fsimage.

          Show
          Raghu Angadi added a comment - There is another more immediate issue with the patch. If a file is started but is not closed, when NameNode is restarted, it appears as a directory while loading the fsimage. Currently if number of blocks is zero, then it is treated as a directory in while loading fsimage. Should we file another jira for it? I was working with patch for HADOOP-1298 which has directory specific fields, I saw for the first time an EOF error while loading the fsimage.
          Hide
          dhruba borthakur added a comment -

          I have a patch for the above-mentioned problem. Please see HADOOP-1887.

          Show
          dhruba borthakur added a comment - I have a patch for the above-mentioned problem. Please see HADOOP-1887 .
          Hide
          dhruba borthakur added a comment -

          Erroneously opened this issue. Follow-up work in HADOOP-1887.

          Show
          dhruba borthakur added a comment - Erroneously opened this issue. Follow-up work in HADOOP-1887 .
          Hide
          Raghu Angadi added a comment -

          Dhruba, HADOOP-1887 is not related to my above comment. But you might have fixed both in HADOOP-1887.

          Show
          Raghu Angadi added a comment - Dhruba, HADOOP-1887 is not related to my above comment. But you might have fixed both in HADOOP-1887 .
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-Nightly #256 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/256/ )

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development