Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6264

Provide FileSystem#create() variant which throws exception if parent directory doesn't exist

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:

      Description

      FileSystem#createNonRecursive() is deprecated.
      However, there is no DistributedFileSystem#create() implementation which throws exception if parent directory doesn't exist.
      This limits clients' migration away from the deprecated method.

      For HBase, IO fencing relies on the behavior of FileSystem#createNonRecursive().

      Variant of create() method should be added which throws exception if parent directory doesn't exist.

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          Another option is to un-deprecate FileSystem#createNonRecursive().

          Show
          Ted Yu added a comment - Another option is to un-deprecate FileSystem#createNonRecursive().
          Hide
          Ted Yu added a comment -

          Raising priority to Major due to client unable to migrate off deprecated API.

          Show
          Ted Yu added a comment - Raising priority to Major due to client unable to migrate off deprecated API.
          Hide
          Enis Soztutar added a comment -

          Correction: For HBase, it is not only a unit test that requires createNonRecursive(). It is pretty important for doing fencing properly for the dead region servers. See my comments at https://issues.apache.org/jira/browse/HBASE-11045?focusedCommentId=13977198&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13977198

          It is unfortunate that createNonRecursive() was deprecated without a proper replacement. HBase relies on atomic createNonRecursive for semantic correctness and prevent data loss via doing fencing by relying on atomic rename + createNonRecursive().

          Show
          Enis Soztutar added a comment - Correction: For HBase, it is not only a unit test that requires createNonRecursive(). It is pretty important for doing fencing properly for the dead region servers. See my comments at https://issues.apache.org/jira/browse/HBASE-11045?focusedCommentId=13977198&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13977198 It is unfortunate that createNonRecursive() was deprecated without a proper replacement. HBase relies on atomic createNonRecursive for semantic correctness and prevent data loss via doing fencing by relying on atomic rename + createNonRecursive().
          Hide
          Ted Yu added a comment -

          Patch v1 un-deprecates createNonRecursive().

          Show
          Ted Yu added a comment - Patch v1 un-deprecates createNonRecursive().
          Hide
          Steve Loughran added a comment -

          If we're targeting 2.5.0 here, I wouldn't do it without actually implementing this operation for most filesystems, because for a real FS it is going to be more atomic than mkdirs+create; blobstores are special - which is why HADOOP-9565 proposes a marker saying "abandon all expectations"

          Most of the existing implementations of FileSystem.create() should be able to call their internal createNonRecursive() implementation after doing the mkdirs op

          Show
          Steve Loughran added a comment - If we're targeting 2.5.0 here, I wouldn't do it without actually implementing this operation for most filesystems, because for a real FS it is going to be more atomic than mkdirs+create; blobstores are special - which is why HADOOP-9565 proposes a marker saying "abandon all expectations" Most of the existing implementations of FileSystem.create() should be able to call their internal createNonRecursive() implementation after doing the mkdirs op
          Hide
          Ted Yu added a comment -

          I wonder if sub-task(s) should be created according to Steve's comment above.

          Show
          Ted Yu added a comment - I wonder if sub-task(s) should be created according to Steve's comment above.

            People

            • Assignee:
              Unassigned
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development