Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3107 HDFS truncate
  3. HDFS-7677

DistributedFileSystem#truncate should resolve symlinks

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We should resolve the symlinks in DistributedFileSystem#truncate as we do for create, open, append and so on, I don't see any reason not support it.

      1. HDFS-7677.001.patch
        3 kB
        Yi Liu
      2. HDFS-7677.002.patch
        3 kB
        Yi Liu

        Activity

        Hide
        hitliuyi Yi Liu added a comment - - edited

        Konstantin Shvachko, also I notice you mention in snapshot with truncate design doc:

        Truncate for HDFS symlink is not supported

        I don't see any reason not support, any thought?

        Show
        hitliuyi Yi Liu added a comment - - edited Konstantin Shvachko , also I notice you mention in snapshot with truncate design doc: Truncate for HDFS symlink is not supported I don't see any reason not support, any thought?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12694546/HDFS-7677.001.patch
        against trunk revision 7b82c4a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
        org.apache.hadoop.hdfs.TestDecommission
        org.apache.hadoop.hdfs.server.datanode.TestBlockScanner

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9328//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9328//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12694546/HDFS-7677.001.patch against trunk revision 7b82c4a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.datanode.TestBlockScanner Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9328//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9328//console This message is automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        I don't see it in my design, which is this one
        https://issues.apache.org/jira/secure/attachment/12679075/HDFS_truncate.pdf

        I don't see reasons not supporting it either.

        Show
        shv Konstantin Shvachko added a comment - I don't see it in my design, which is this one https://issues.apache.org/jira/secure/attachment/12679075/HDFS_truncate.pdf I don't see reasons not supporting it either.
        Hide
        cmccabe Colin P. McCabe added a comment -

        I think we should support this, just for consistency with the other operations. But keep in mind that symlinks are hard-disabled in branch-2 because of security and other issues, and that's probably not going to change any time soon. See HADOOP-10019 for details.

        Show
        cmccabe Colin P. McCabe added a comment - I think we should support this, just for consistency with the other operations. But keep in mind that symlinks are hard-disabled in branch-2 because of security and other issues, and that's probably not going to change any time soon. See HADOOP-10019 for details.
        Hide
        hitliuyi Yi Liu added a comment -

        I mean in the design doc of HDFS-7056.
        Thanks for your confirmation about supporting it.

        Show
        hitliuyi Yi Liu added a comment - I mean in the design doc of HDFS-7056 . Thanks for your confirmation about supporting it.
        Hide
        hitliuyi Yi Liu added a comment -

        That's great. So Konstantin Shvachko and Colin P. McCabe please help to take a look at the patch, thanks.

        Show
        hitliuyi Yi Liu added a comment - That's great. So Konstantin Shvachko and Colin P. McCabe please help to take a look at the patch, thanks.
        Hide
        shv Konstantin Shvachko added a comment -

        Looks good. Few minor nits on the test:

        1. Could you move testTruncate4Symlink() just before writeContents(). It is positioned now between two testSnapshotWithAppendTruncate() mehods, which should logically be together.
        2. Looks like you truncate on the block boundary, so
          if(!isReady) { ...
          can be replaced with
          assertTrue("Recovery is not expected.", isReady);
        3. Replacing AppendTestUtil.checkFullFile() with checkFullFile() would save a few bytes of code.
        Show
        shv Konstantin Shvachko added a comment - Looks good. Few minor nits on the test: Could you move testTruncate4Symlink() just before writeContents() . It is positioned now between two testSnapshotWithAppendTruncate() mehods, which should logically be together. Looks like you truncate on the block boundary, so if(!isReady) { ... can be replaced with assertTrue("Recovery is not expected.", isReady); Replacing AppendTestUtil.checkFullFile() with checkFullFile() would save a few bytes of code.
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Konstantin Shvachko! Update the patch to address comments.

        Show
        hitliuyi Yi Liu added a comment - Thanks Konstantin Shvachko ! Update the patch to address comments.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12694736/HDFS-7677.002.patch
        against trunk revision 6f9fe76.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9341//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9341//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12694736/HDFS-7677.002.patch against trunk revision 6f9fe76. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9341//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9341//console This message is automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        +1 from me too.
        Don't think this could do anything to eclipse:eclipse.

        Show
        shv Konstantin Shvachko added a comment - +1 from me too. Don't think this could do anything to eclipse:eclipse.
        Hide
        hitliuyi Yi Liu added a comment -

        Committed to trunk and branch-2. Thanks Konstantin Shvachko for review, also thanks Colin P. McCabe for comments.

        Show
        hitliuyi Yi Liu added a comment - Committed to trunk and branch-2. Thanks Konstantin Shvachko for review, also thanks Colin P. McCabe for comments.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6947 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6947/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6947 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6947/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #87 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/87/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #87 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/87/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #821 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/821/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #821 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/821/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #84 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/84/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #84 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/84/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2019 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2019/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2019 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2019/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #88 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/88/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #88 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/88/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2038/)
        HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2038/ ) HDFS-7677 . DistributedFileSystem#truncate should resolve symlinks. (yliu) (yliu: rev 9ca565e9704d236ce839c0138d82d54453d793fb) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            hitliuyi Yi Liu
            Reporter:
            hitliuyi Yi Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development