Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3107 HDFS truncate
  3. HDFS-7843

A truncated file is corrupted after rollback from a rolling upgrade

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Here is a rolling upgrade truncate test from Brandon Li. The basic test step is: (3 nodes cluster with HA)
      1. upload a file to hdfs
      2. start rollingupgrade. finish rollingupgrade for namenode and one datanode.
      3. truncate the file in hdfs to 1byte
      4. do rollback
      5. download file from hdfs, check file size to be original size

      I see the file size in hdfs is correct but can't read it because the block is corrupted.

      1. h7843_20150226.patch
        6 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        About to make an RC for 2.6.1. Dropping the 2.6.1 label as truncate wasn't there in 2.6.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - About to make an RC for 2.6.1. Dropping the 2.6.1 label as truncate wasn't there in 2.6.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Adding back 2.6.1-candidate label for tracking purposes. Will remove it once 2.6.1 is done.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Adding back 2.6.1-candidate label for tracking purposes. Will remove it once 2.6.1 is done.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Removing the 2.6.1-candidate label as this is not applicable to 2.6.0.

        Show
        sjlee0 Sangjin Lee added a comment - Removing the 2.6.1-candidate label as this is not applicable to 2.6.0.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2066 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2066/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2066 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2066/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/116/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/116/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/107/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/107/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2048 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2048/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2048 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2048/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #850 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/850/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #850 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/850/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/116/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/116/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7202 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7202/)
        HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7202 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7202/ ) HDFS-7843 . A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo: rev 606f5b517ffbeae0140a8c80b4cddc012c7fb3c4) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Thanks Brandon for reviewing the patch.

        I have committed this.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Brandon for reviewing the patch. I have committed this.
        Hide
        brandonli Brandon Li added a comment -

        +1. The patch looks good to me.

        Show
        brandonli Brandon Li added a comment - +1. The patch looks good to me.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        All the failed tests are not related to the patch. They somehow timed out.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - All the failed tests are not related to the patch. They somehow timed out.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12700791/h7843_20150226.patch
        against trunk revision 1a68fc4.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDecommission
        org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
        org.apache.hadoop.hdfs.qjournal.TestNNWithQJM

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9664//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9664//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12700791/h7843_20150226.patch against trunk revision 1a68fc4. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.namenode.TestFileTruncate org.apache.hadoop.hdfs.qjournal.TestNNWithQJM Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9664//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9664//console This message is automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h7843_20150226.patch: use copy-on-truncate for rolling upgrade.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h7843_20150226.patch: use copy-on-truncate for rolling upgrade.

          People

          • Assignee:
            szetszwo Tsz Wo Nicholas Sze
            Reporter:
            szetszwo Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development