Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Similar with append, lazy persist (memory) file should not support truncate currently. Quote the reason from HDFS-6581 design doc:

      Appends to files created with the LAZY_PERSISTflag will not be allowed in the initial implementation to avoid the complexity of keeping in­memory and on­disk replicas in sync on a given DataNode.

      1. HDFS-7634.001.patch
        1 kB
        Yi Liu
      2. HDFS-7634.002.patch
        1 kB
        Yi Liu

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692748/HDFS-7634.001.patch
          against trunk revision 000ca83.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9247//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9247//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692748/HDFS-7634.001.patch against trunk revision 000ca83. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9247//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9247//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Looks good. Only you don't need to assign file twice. You first assign it using asFile(), then using INodeFile.valueOf().

          Show
          shv Konstantin Shvachko added a comment - Looks good. Only you don't need to assign file twice. You first assign it using asFile(), then using INodeFile.valueOf().
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Konstantin Shvachko for review. You are right, the first assign is unnecessary and I was going to remove it. It was removed as part of HDFS-7638.

          So we just need to rebase the patch, please look at the new patch, thanks.

          Show
          hitliuyi Yi Liu added a comment - Thanks Konstantin Shvachko for review. You are right, the first assign is unnecessary and I was going to remove it. It was removed as part of HDFS-7638 . So we just need to rebase the patch, please look at the new patch, thanks.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693203/HDFS-7634.002.patch
          against trunk revision 5a6c084.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9273//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9273//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693203/HDFS-7634.002.patch against trunk revision 5a6c084. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9273//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9273//console This message is automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the patch, thanks for catching this Yi Liu.

          Konstantin Shvachko if you have no objections I can commit it today evening. Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the patch, thanks for catching this Yi Liu . Konstantin Shvachko if you have no objections I can commit it today evening. Thanks.
          Hide
          shv Konstantin Shvachko added a comment -

          No objections. +1

          Show
          shv Konstantin Shvachko added a comment - No objections. +1
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I filed HDFS-7643 to add a test case for this issue. I will commit the patch shortly.

          Show
          arpitagarwal Arpit Agarwal added a comment - I filed HDFS-7643 to add a test case for this issue. I will commit the patch shortly.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the review Konstantin Shvachko. Committed to trunk.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the review Konstantin Shvachko . Committed to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #6894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6894/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6894/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks a lot for Arpit Agarwal and Konstantin Shvachko for the review and commit.

          Show
          hitliuyi Yi Liu added a comment - Thanks a lot for Arpit Agarwal and Konstantin Shvachko for the review and commit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #814 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/814/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #814 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/814/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #80 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/80/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #80 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/80/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #77 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/77/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #77 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/77/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2012 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2012/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2012 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2012/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #81 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/81/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #81 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/81/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2031/)
          HDFS-7634. Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            HDFS-7634. Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2031/ ) HDFS-7634 . Disallow truncation of Lazy persist files. (Contributed by Yi Liu) (arp: rev c09c65b2125908855a5f1d0047bc164ea4bea04d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java HDFS-7634 . Fix CHANGES.txt (arp: rev dd0228b8f7d9b3851aa408398eef516b93522e95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          shv Konstantin Shvachko added a comment -

          Merged to branch-2.

          Show
          shv Konstantin Shvachko added a comment - Merged to branch-2.

            People

            • Assignee:
              hitliuyi Yi Liu
              Reporter:
              hitliuyi Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development