Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The test uses <RegexAcrossOutputComparator> in a single regex, which does not match on Windows as shown below.

      2015-04-30 01:14:01,737 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(155)) - -------------------------------------------
      2015-04-30 01:14:01,737 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(156)) -                     Test ID: [31]
      2015-04-30 01:14:01,737 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(157)) -            Test Description: [help: help for find]
      2015-04-30 01:14:01,737 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(158)) - 
      2015-04-30 01:14:01,738 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(162)) -               Test Commands: [-help find]
      2015-04-30 01:14:01,738 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(166)) - 
      2015-04-30 01:14:01,738 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(173)) - 
      2015-04-30 01:14:01,738 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(177)) -                  Comparator: [RegexpAcrossOutputComparator]
      2015-04-30 01:14:01,738 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(179)) -          Comparision result:   [fail]
      2015-04-30 01:14:01,739 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(181)) -             Expected output:   [-find <path> \.\.\. <expression> \.\.\. :
        Finds all files that match the specified expression and
        applies selected actions to them\. If no <path> is specified
        then defaults to the current working directory\. If no
        expression is specified then defaults to -print\.
        
        The following primary expressions are recognised:
          -name pattern
          -iname pattern
            Evaluates as true if the basename of the file matches the
            pattern using standard file system globbing\.
            If -iname is used then the match is case insensitive\.
        
          -print
          -print0
            Always evaluates to true. Causes the current pathname to be
            written to standard output followed by a newline. If the -print0
            expression is used then an ASCII NULL character is appended rather
            than a newline.
        
        The following operators are recognised:
          expression -a expression
          expression -and expression
          expression expression
            Logical AND operator for joining two expressions\. Returns
            true if both child expressions return true\. Implied by the
            juxtaposition of two expressions and so does not need to be
            explicitly specified\. The second expression will not be
            applied if the first fails\.
      ]
      2015-04-30 01:14:01,739 INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(183)) -               Actual output:   [-find <path> ... <expression> ... :
        Finds all files that match the specified expression and
        applies selected actions to them. If no <path> is specified
        then defaults to the current working directory. If no
        expression is specified then defaults to -print.
        
        The following primary expressions are recognised:
          -name pattern
          -iname pattern
            Evaluates as true if the basename of the file matches the
            pattern using standard file system globbing.
            If -iname is used then the match is case insensitive.
        
          -print
          -print0
            Always evaluates to true. Causes the current pathname to be
            written to standard output followed by a newline. If the -print0
            expression is used then an ASCII NULL character is appended rather
            than a newline.
        
        The following operators are recognised:
          expression -a expression
          expression -and expression
          expression expression
            Logical AND operator for joining two expressions. Returns
            true if both child expressions return true. Implied by the
            juxtaposition of two expressions and so does not need to be
            explicitly specified. The second expression will not be
            applied if the first fails.
      ]
      
      1. HDFS-8310-002.patch
        1 kB
        Kiran Kumar M R
      2. HDFS-8310-001.patch
        1 kB
        Kiran Kumar M R

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2136/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2136/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/187/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/187/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/177/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/177/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2118 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2118/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2118 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2118/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #920 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/920/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #920 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/920/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/187/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/187/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7746/)
        HDFS-8310. Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7746/ ) HDFS-8310 . Fix TestCLI.testAll 'help: help for find' on Windows. (Kiran Kumar M R via Xiaoyu Yao) (xyao: rev 7a26d174aff9535f7a60711bee586e225891b383) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/util/RegexpAcrossOutputComparator.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Kiran Kumar M R for the contribution and Akira Ajisaka for the review. I've committed the patch to trunk and branch-2.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Kiran Kumar M R for the contribution and Akira Ajisaka for the review. I've committed the patch to trunk and branch-2.
        Hide
        xyao Xiaoyu Yao added a comment -

        +1 for v002 patch. I will commit it to trunk and branch-2 shortly.

        Show
        xyao Xiaoyu Yao added a comment - +1 for v002 patch. I will commit it to trunk and branch-2 shortly.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1

        Show
        ajisakaa Akira Ajisaka added a comment - +1
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 9s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 3s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 22m 40s Tests passed in hadoop-common.
            40m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730773/HDFS-8310-002.patch
        Optional Tests javac unit findbugs checkstyle
        git revision trunk / a583a40
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/10830/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10830/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10830/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 40s Tests passed in hadoop-common.     40m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730773/HDFS-8310-002.patch Optional Tests javac unit findbugs checkstyle git revision trunk / a583a40 hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/10830/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10830/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10830/console This message was automatically generated.
        Hide
        kiranmr Kiran Kumar M R added a comment -

        Thanks for review Xiaoyu, I have added space and attached patch.

        Show
        kiranmr Kiran Kumar M R added a comment - Thanks for review Xiaoyu, I have added space and attached patch.
        Hide
        xyao Xiaoyu Yao added a comment -

        Kiran Kumar M R, I just found one NIT before commit it: It seems a space is missed between if and '(', and the '(' the '{' below. Can you update the patch?

        if(Shell.WINDOWS){
        
        Show
        xyao Xiaoyu Yao added a comment - Kiran Kumar M R , I just found one NIT before commit it: It seems a space is missed between if and '(', and the '(' the '{' below. Can you update the patch? if (Shell.WINDOWS){
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 11s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 5s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 23m 4s Tests passed in hadoop-common.
            40m 59s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729984/HDFS-8310-001.patch
        Optional Tests javac unit findbugs checkstyle
        git revision trunk / a319771
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/10754/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10754/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10754/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 5s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 4s Tests passed in hadoop-common.     40m 59s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729984/HDFS-8310-001.patch Optional Tests javac unit findbugs checkstyle git revision trunk / a319771 hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/10754/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10754/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10754/console This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Kiran Kumar M R for fixing this and Brahma Reddy Battula for the review.
        +1 pending Jenkins.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Kiran Kumar M R for fixing this and Brahma Reddy Battula for the review. +1 pending Jenkins.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Kiran Kumar M R thanks for taking this issue..Patch LGTM,+1 ( non binding)

        Show
        brahmareddy Brahma Reddy Battula added a comment - Kiran Kumar M R thanks for taking this issue..Patch LGTM,+1 ( non binding)
        Hide
        kiranmr Kiran Kumar M R added a comment -

        Test case failure is due to CR(\r) carriage return character in new line. Output of command execution "hadoop fs -help find" on windows is having both \r\n characters in new line. Expected output string is taken from testConf.xml, this will only have \n characters for new line. XML Parser (SAX) will normalize new line on all platforms.

        Due to this difference in newline characters, test output comparison is failing.

        RegexpAcrossOutputComparator was intended for comparing multi-line outputs. I have modified it to cleanup \r from input parameters before doing regex comparison.

            if(Shell.WINDOWS){
              actual = actual.replaceAll("\\r", "");
              expected = expected.replaceAll("\\r", "");
            }
        

        After this modification test cases are passing,
        Review the fix.

        Show
        kiranmr Kiran Kumar M R added a comment - Test case failure is due to CR(\r) carriage return character in new line. Output of command execution "hadoop fs -help find" on windows is having both \r\n characters in new line. Expected output string is taken from testConf.xml, this will only have \n characters for new line. XML Parser (SAX) will normalize new line on all platforms. Due to this difference in newline characters, test output comparison is failing. RegexpAcrossOutputComparator was intended for comparing multi-line outputs. I have modified it to cleanup \r from input parameters before doing regex comparison. if (Shell.WINDOWS){ actual = actual.replaceAll( "\\r" , ""); expected = expected.replaceAll( "\\r" , ""); } After this modification test cases are passing, Review the fix.

          People

          • Assignee:
            kiranmr Kiran Kumar M R
            Reporter:
            xyao Xiaoyu Yao
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development