Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We'd like to add a new feature hardlink to HDFS that allows harlinked files to share data without copying. Currently we will support hardlinking only closed files, but it could be extended to unclosed files as well.

      Among many potential use cases of the feature, the following two are primarily used in facebook:
      1. This provides a lightweight way for applications like hbase to create a snapshot;
      2. This also allows an application like Hive to move a table to a different directory without breaking current running hive queries.

      1. HDFS-HardLink.pdf
        1.03 MB
        Liyin Tang

        Issue Links

          Activity

          Hairong Kuang created issue -
          Liyin Tang made changes -
          Field Original Value New Value
          Link This issue is required by HBASE-5940 [ HBASE-5940 ]
          Hairong Kuang made changes -
          Description We'd like to add a new feature to HDFS, hardlink, that allows harlinked files to share data without copying. Currently we will support hardlinking only closed files, but it could be extended to unclosed files as well.

          Among many potential use cases of feature, the following two are primarily used in facebook:
          1. This provides a lightweight way for applications like hbase to create a snapshot;
          2. This also allows an application like Hive to move a table to a different directory without breaking current running hive queries.
          We'd like to add a new feature hardlink to HDFS that allows harlinked files to share data without copying. Currently we will support hardlinking only closed files, but it could be extended to unclosed files as well.

          Among many potential use cases of the feature, the following two are primarily used in facebook:
          1. This provides a lightweight way for applications like hbase to create a snapshot;
          2. This also allows an application like Hive to move a table to a different directory without breaking current running hive queries.
          Liyin Tang made changes -
          Attachment HDFS-HardLinks.pdf [ 12525672 ]
          Liyin Tang made changes -
          Attachment HDFS-HardLinks.pdf [ 12525672 ]
          Liyin Tang made changes -
          Attachment HDFS-HardLink.pdf [ 12526109 ]
          Liyin Tang made changes -
          Attachment HDFS-HardLink.pdf [ 12526109 ]
          Liyin Tang made changes -
          Attachment HDFS-HardLink.pdf [ 12526189 ]
          Konstantin Shvachko made changes -
          Assignee Liyin Tang [ liyin ] Konstantin Shvachko [ shv ]
          Konstantin Shvachko made changes -
          Assignee Konstantin Shvachko [ shv ] Liyin Tang [ liyin ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3154 [ HDFS-3154 ]

            People

            • Assignee:
              Liyin Tang
              Reporter:
              Hairong Kuang
            • Votes:
              3 Vote for this issue
              Watchers:
              59 Start watching this issue

              Dates

              • Created:
                Updated:

                Development