Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.7.0
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Modify TestCLI to include general truncate tests.

      1. HDFS-7058.patch
        2 kB
        Dasha Boudnik
      2. HDFS-7058.patch
        3 kB
        Dasha Boudnik
      3. HDFS-7058.patch
        3 kB
        Dasha Boudnik
      4. HDFS-7058.patch
        4 kB
        Dasha Boudnik

        Activity

        Hide
        shv Konstantin Shvachko added a comment -

        Some tests I see should be included:

        • general truncate in TestCLI
        • truncate on the boundary
        • truncate to 0 length
        • Regular recovery
        • Restart of NN while recovery is in progress
        • snpashots + trancate
        Show
        shv Konstantin Shvachko added a comment - Some tests I see should be included: general truncate in TestCLI truncate on the boundary truncate to 0 length Regular recovery Restart of NN while recovery is in progress snpashots + trancate
        Hide
        shv Konstantin Shvachko added a comment -

        All tests except TestCLI are already included in the unit tests under HDFS-3107 and HDFS-7056.
        So this is only about TestCLI now unless something additional comes up.

        Show
        shv Konstantin Shvachko added a comment - All tests except TestCLI are already included in the unit tests under HDFS-3107 and HDFS-7056 . So this is only about TestCLI now unless something additional comes up.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Revised summary and description.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Revised summary and description.
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        I can look into this. Thanks!

        Show
        dasha.boudnik Dasha Boudnik added a comment - I can look into this. Thanks!
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        Patch attached. I've included four variations of truncate:

        • truncate to small size
        • truncate to large (not much smaller than original file) size
        • truncate to size greater than actual file
        • truncate without waiting for block recovery

        When leaving out -w option (see last point above), clean-up cannot be performed without disrupting the test, so, instead, a directory with a name that is unlikely to be used by any other tests is created and left there. If anyone would suggest using a timestamp as the dir name instead, let me know; I didn't do that to avoid messing with TestCLI.java.

        Please let me know if I've missed any other ways to test truncate that we should include in this patch. Thanks!

        Show
        dasha.boudnik Dasha Boudnik added a comment - Patch attached. I've included four variations of truncate: truncate to small size truncate to large (not much smaller than original file) size truncate to size greater than actual file truncate without waiting for block recovery When leaving out -w option (see last point above), clean-up cannot be performed without disrupting the test, so, instead, a directory with a name that is unlikely to be used by any other tests is created and left there. If anyone would suggest using a timestamp as the dir name instead, let me know; I didn't do that to avoid messing with TestCLI.java. Please let me know if I've missed any other ways to test truncate that we should include in this patch. Thanks!
        Hide
        shv Konstantin Shvachko added a comment -

        I would do it as in append test.

        1. Use /user/USERNAME/dir? as a directory
        2. Don't need NAMENODE in the path as it is already in the command
        3. We should -cat after truncating to make sure the length is right
        Show
        shv Konstantin Shvachko added a comment - I would do it as in append test. Use /user/USERNAME/dir? as a directory Don't need NAMENODE in the path as it is already in the command We should -cat after truncating to make sure the length is right
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        Konstantin Shvachko, thanks. I've removed truncate to a large file length since the comparators we're using can't check multiple lines when running -cat on the output. What's left:

        • truncate to 5 bytes without waiting for block recovery
        • truncate to 5 bytes after waiting for block recovery
        • attempt to truncate to length greater than length of actual file
        Show
        dasha.boudnik Dasha Boudnik added a comment - Konstantin Shvachko , thanks. I've removed truncate to a large file length since the comparators we're using can't check multiple lines when running -cat on the output. What's left: truncate to 5 bytes without waiting for block recovery truncate to 5 bytes after waiting for block recovery attempt to truncate to length greater than length of actual file
        Hide
        shv Konstantin Shvachko added a comment -

        Dasha, this looks good to me.
        You should click "submit patch" to start jenkins build. But before you do you will probably need to re-attach the patch.
        The best way is to use git diff --no-prefix. Jenkins will most likely not be able to apply your patch as is. I couldn't without editing.

        Show
        shv Konstantin Shvachko added a comment - Dasha, this looks good to me. You should click "submit patch" to start jenkins build. But before you do you will probably need to re-attach the patch. The best way is to use git diff --no-prefix . Jenkins will most likely not be able to apply your patch as is. I couldn't without editing.
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        Thanks, Konstantin. Patch attached, will submit in a second.

        Show
        dasha.boudnik Dasha Boudnik added a comment - Thanks, Konstantin. Patch attached, will submit in a second.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12697365/HDFS-7058.patch
        against trunk revision 1382ae5.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697365/HDFS-7058.patch against trunk revision 1382ae5. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9497//console This message is automatically generated.
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        I've checked the findbugs and failed test warnings, and they have nothing to do with the changes I've made. Ready to commit, as far as I can tell.

        Show
        dasha.boudnik Dasha Boudnik added a comment - I've checked the findbugs and failed test warnings, and they have nothing to do with the changes I've made. Ready to commit, as far as I can tell.
        Hide
        shv Konstantin Shvachko added a comment -

        +1
        TestRetryCacheWithHA failure tracked in HDFS-7524. Findbug is unrelated.

        Show
        shv Konstantin Shvachko added a comment - +1 TestRetryCacheWithHA failure tracked in HDFS-7524 . Findbug is unrelated.
        Hide
        zero45 Plamen Jeliazkov added a comment -

        The 1st and 3rd test are the same thing, no?

        Show
        zero45 Plamen Jeliazkov added a comment - The 1st and 3rd test are the same thing, no?
        Hide
        dasha.boudnik Dasha Boudnik added a comment -

        So, because it takes time to truncate the file, the test doesn't pass in CLI if you don't wait for block recovery – it doesn't get truncated quickly enough. I excluded truncate without -w altogether from this patch. Unless there's a way around this, I guess can submit it?

        Show
        dasha.boudnik Dasha Boudnik added a comment - So, because it takes time to truncate the file, the test doesn't pass in CLI if you don't wait for block recovery – it doesn't get truncated quickly enough. I excluded truncate without -w altogether from this patch. Unless there's a way around this, I guess can submit it?
        Hide
        zero45 Plamen Jeliazkov added a comment -

        I understand. Yes was best to remove it. +1

        Show
        zero45 Plamen Jeliazkov added a comment - I understand. Yes was best to remove it. +1
        Hide
        shv Konstantin Shvachko added a comment -

        I just committed this. Thank you, Dasha.

        Show
        shv Konstantin Shvachko added a comment - I just committed this. Thank you, Dasha.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7060 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7060/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7060 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7060/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12697620/HDFS-7058.patch
        against trunk revision 260b5e3.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697620/HDFS-7058.patch against trunk revision 260b5e3. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9509//console This message is automatically generated.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #100 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/100/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #100 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/100/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/834/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/834/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2032 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2032/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2032 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2032/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/101/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/101/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2051/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2051/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/)
        HDFS-7058. Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/ ) HDFS-7058 . Tests for truncate CLI. Contributed by Dasha Boudnik. (shv: rev b73956fdad87e03e9875a96465579a1a3b17c498) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            dasha.boudnik Dasha Boudnik
            Reporter:
            shv Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development