Hadoop Common
  1. Hadoop Common
  2. HADOOP-726

HDFS locking mechanisms should be simplified or removed

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      HDFS includes a locking mechanism that allows clients to lock files and directories in the filesystem. This requires that the Namenode, in addition to all other management of filesystem state also has to act like a lock manager on behalf of clients. Rather than burden the Namenode with this Hadoop should include a distinct general purpose distributed lock manager.

      In the interim, the locking functionality on the filesystem should be simplified to permit only file locks or removed altogether. At this point, removing or simplifying the functionality is not likely to break client code, removing it later will be much harder. In any case, locks in HDFS don't work very well, see HADOOP-656. Also, directory locks are overkill most UNIX fses don't support it.

      1. remove-fs-locks.patch
        8 kB
        Raghu Angadi
      2. HADOOP-726-2.patch
        6 kB
        Raghu Angadi
      3. HADOOP-726-1.patch
        5 kB
        Raghu Angadi

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          Attached remove-fs-locks.patch. It is not meant for inclusion. Initially we just want to deprecate the locking interface and remove the functionality later. Before deprecating I removed the locks to verify hadoop passes build tests.

          I will submit another patch that deprecates the inteface.

          Show
          Raghu Angadi added a comment - Attached remove-fs-locks.patch. It is not meant for inclusion. Initially we just want to deprecate the locking interface and remove the functionality later. Before deprecating I removed the locks to verify hadoop passes build tests. I will submit another patch that deprecates the inteface.
          Hide
          Raghu Angadi added a comment -

          HADOOP-726.patch marks locking functions @Deprecated.

          Show
          Raghu Angadi added a comment - HADOOP-726 .patch marks locking functions @Deprecated.
          Hide
          Hadoop QA added a comment -

          +1, because http://issues.apache.org/jira/secure/attachment/12349472/HADOOP-726-1.patch applied and successfully tested against trunk revision r498829.

          Show
          Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12349472/HADOOP-726-1.patch applied and successfully tested against trunk revision r498829.
          Hide
          Doug Cutting added a comment -

          I think what's most important is to deprecate the public methods. You've deprecated a number of package-private methods, and left some prominent public methods, like FileSystem#lock(), S3FileSystem#lock() and LocalFileSystem#lock(). These should also be deprecated. A separate issue should be created that removes these methods and their implementations in the next release.

          Show
          Doug Cutting added a comment - I think what's most important is to deprecate the public methods. You've deprecated a number of package-private methods, and left some prominent public methods, like FileSystem#lock(), S3FileSystem#lock() and LocalFileSystem#lock(). These should also be deprecated. A separate issue should be created that removes these methods and their implementations in the next release.
          Hide
          Raghu Angadi added a comment -

          Thanks Doug. 2.patch depcrecates the public lock / release() interface for FileSystem and other related classes. verified that this does not introduce any deprecation warning in compiler.

          I will file another issue for removing the functionality.

          Show
          Raghu Angadi added a comment - Thanks Doug. 2.patch depcrecates the public lock / release() interface for FileSystem and other related classes. verified that this does not introduce any deprecation warning in compiler. I will file another issue for removing the functionality.
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Raghu!

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Raghu!

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Sameer Paranjpye
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development