Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12657

Add a option to skip newline on empty files with getMerge -nl

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0, 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added -skip-empty-file option to hadoop fs -getmerge command. With the option, delimiter (LF) is not printed for empty files even if -nl option is used.

      Description

      Hello everyone,

      I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards.

      In this situation it is maybe good to have another argument that allows skipping empty files.
      Thing one could try to implement this feature:

      The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
      return the number of bytes copied which would be convenient as one could
      skip append the new line when 0 bytes where copied or one would check the file size before.

      I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way.

      1. HDFS-8836-07.patch
        8 kB
        Kanaka Kumar Avvaru
      2. HDFS-8836-06.patch
        8 kB
        Kanaka Kumar Avvaru
      3. HDFS-8836-05.patch
        7 kB
        Kanaka Kumar Avvaru
      4. HDFS-8836-04.patch
        7 kB
        Kanaka Kumar Avvaru
      5. HDFS-8836-03.patch
        7 kB
        Kanaka Kumar Avvaru
      6. HDFS-8836-02.patch
        7 kB
        Kanaka Kumar Avvaru
      7. HDFS-8836-01.patch
        6 kB
        Kanaka Kumar Avvaru

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 24s trunk passed
        +1 compile 8m 10s trunk passed with JDK v1.8.0_66
        +1 compile 8m 34s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 47s trunk passed
        +1 javadoc 0m 52s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 1s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 41s the patch passed
        +1 compile 7m 58s the patch passed with JDK v1.8.0_66
        +1 javac 7m 58s the patch passed
        +1 compile 8m 43s the patch passed with JDK v1.7.0_91
        +1 javac 8m 43s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 3s the patch passed with JDK v1.7.0_91
        -1 unit 6m 59s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 7m 24s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        68m 55s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestIPC



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777652/HDFS-8836-07.patch
        JIRA Issue HADOOP-12657
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux ee783aa3ae64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bd5e207
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 8m 10s trunk passed with JDK v1.8.0_66 +1 compile 8m 34s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 1s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 41s the patch passed +1 compile 7m 58s the patch passed with JDK v1.8.0_66 +1 javac 7m 58s the patch passed +1 compile 8m 43s the patch passed with JDK v1.7.0_91 +1 javac 8m 43s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_91 -1 unit 6m 59s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 7m 24s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 68m 55s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777652/HDFS-8836-07.patch JIRA Issue HADOOP-12657 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ee783aa3ae64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bd5e207 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8272/console This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8994/)
        HADOOP-12657. Add a option to skip newline on empty files with getMerge (aajisaka: rev 061c05cc05ff6257b14c5c4f25cbcec2d184cda7)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
        • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
        • hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8994/ ) HADOOP-12657 . Add a option to skip newline on empty files with getMerge (aajisaka: rev 061c05cc05ff6257b14c5c4f25cbcec2d184cda7) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md hadoop-common-project/hadoop-common/src/test/resources/testConf.xml hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        ajisakaa Akira Ajisaka added a comment -

        and thanks Jan Filipiak for reviewing the patches!

        Show
        ajisakaa Akira Ajisaka added a comment - and thanks Jan Filipiak for reviewing the patches!
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk, branch-2, and branch-2.8. Thanks Kanaka Kumar Avvaru for the contribution!

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks Kanaka Kumar Avvaru for the contribution!
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, the test failure was reported at HADOOP-12588 and was (hopefully) fixed.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, the test failure was reported at HADOOP-12588 and was (hopefully) fixed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 38s trunk passed
        +1 compile 9m 45s trunk passed with JDK v1.8.0_66
        +1 compile 10m 10s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 1m 16s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 2m 12s trunk passed
        +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 43s the patch passed
        +1 compile 9m 47s the patch passed with JDK v1.8.0_66
        +1 javac 9m 47s the patch passed
        +1 compile 10m 8s the patch passed with JDK v1.7.0_91
        +1 javac 10m 8s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 1m 13s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 24s the patch passed
        +1 javadoc 0m 58s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91
        -1 unit 8m 41s hadoop-common in the patch failed with JDK v1.8.0_66.
        -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        80m 42s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
        JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777652/HDFS-8836-07.patch
        JIRA Issue HDFS-8836
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 111a1f3ee02e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8a4542
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13874/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Max memory used 76MB
        Powered by Apache Yetus 0.1.0 http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13874/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 38s trunk passed +1 compile 9m 45s trunk passed with JDK v1.8.0_66 +1 compile 10m 10s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 12s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 43s the patch passed +1 compile 9m 47s the patch passed with JDK v1.8.0_66 +1 javac 9m 47s the patch passed +1 compile 10m 8s the patch passed with JDK v1.7.0_91 +1 javac 10m 8s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 24s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91 -1 unit 8m 41s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 80m 42s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777652/HDFS-8836-07.patch JIRA Issue HDFS-8836 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 111a1f3ee02e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8a4542 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13874/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13874/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Ok fine. Thanks for the comment Akira Ajisaka. Please review the updated patch. The only change is to make the field private

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Ok fine. Thanks for the comment Akira Ajisaka . Please review the updated patch. The only change is to make the field private
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thank you for your comment.

        I think private and accessors are not required for skipEmptyFileDelimiter as this variable required to be accessed only in this or a subclass.

        I'm thinking the variable should be private because the methods of Merge are not likely to be overridden.

        Show
        ajisakaa Akira Ajisaka added a comment - Thank you for your comment. I think private and accessors are not required for skipEmptyFileDelimiter as this variable required to be accessed only in this or a subclass. I'm thinking the variable should be private because the methods of Merge are not likely to be overridden.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Thanks for taking a look at patch Akira Ajisaka.
        I think private and accessors are not required for skipEmptyFileDelimiter as this variable required to be accessed only in this or a subclass. Also other variables like delimiter doesn't have accessors.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Thanks for taking a look at patch Akira Ajisaka . I think private and accessors are not required for skipEmptyFileDelimiter as this variable required to be accessed only in this or a subclass. Also other variables like delimiter doesn't have accessors.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        LGTM, the test failures look unrelated to the patch. I ran the tests locally and they passed.

        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java:67:23: Variable 'skipEmptyFileDelimiter' must be private and have accessor methods.

        Would you fix the checkstyle issue? I'm +1 if that is addressed.

        Sorry for late response.

        Show
        ajisakaa Akira Ajisaka added a comment - LGTM, the test failures look unrelated to the patch. I ran the tests locally and they passed. ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java:67:23: Variable 'skipEmptyFileDelimiter' must be private and have accessor methods. Would you fix the checkstyle issue? I'm +1 if that is addressed. Sorry for late response.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 11m 9s trunk passed
        +1 compile 17m 49s trunk passed with JDK v1.8.0_66
        +1 compile 14m 17s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 1m 41s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 3m 4s trunk passed
        +1 javadoc 1m 41s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 40s trunk passed with JDK v1.7.0_85
        +1 mvninstall 2m 22s the patch passed
        +1 compile 19m 38s the patch passed with JDK v1.8.0_66
        +1 javac 19m 38s the patch passed
        +1 compile 14m 50s the patch passed with JDK v1.7.0_85
        +1 javac 14m 50s the patch passed
        -1 checkstyle 0m 26s Patch generated 1 new checkstyle issues in hadoop-common-project/hadoop-common (total was 45, now 42).
        +1 mvnsite 1m 45s the patch passed
        +1 mvneclipse 0m 23s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 27s the patch passed
        +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 51s the patch passed with JDK v1.7.0_85
        -1 unit 14m 14s hadoop-common in the patch failed with JDK v1.8.0_66.
        -1 unit 14m 56s hadoop-common in the patch failed with JDK v1.7.0_85.
        +1 asflicense 0m 36s Patch does not generate ASF License warnings.
        130m 14s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.find.TestAnd
          hadoop.metrics2.impl.TestMetricsSystemImpl
          hadoop.ipc.TestDecayRpcScheduler
          hadoop.fs.shell.find.TestPrint
          hadoop.fs.shell.find.TestPrint0
          hadoop.test.TestTimedOutTestsListener
          hadoop.fs.shell.find.TestIname
          hadoop.fs.shell.find.TestName
          hadoop.fs.shell.find.TestFind
          hadoop.ipc.TestRPCWaitForProxy
        JDK v1.7.0_85 Failed junit tests hadoop.fs.TestSymlinkLocalFSFileContext
          hadoop.ha.TestZKFailoverController
          hadoop.metrics2.impl.TestMetricsSystemImpl
          hadoop.fs.shell.find.TestPrint
          hadoop.fs.shell.find.TestPrint0
          hadoop.fs.shell.find.TestIname
          hadoop.fs.shell.find.TestFind
          hadoop.ipc.TestRPCWaitForProxy
          hadoop.ipc.TestIPC



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774577/HDFS-8836-06.patch
        JIRA Issue HDFS-8836
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux d5f994e55d78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9f256d1
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13680/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13680/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 11m 9s trunk passed +1 compile 17m 49s trunk passed with JDK v1.8.0_66 +1 compile 14m 17s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 41s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 40s trunk passed with JDK v1.7.0_85 +1 mvninstall 2m 22s the patch passed +1 compile 19m 38s the patch passed with JDK v1.8.0_66 +1 javac 19m 38s the patch passed +1 compile 14m 50s the patch passed with JDK v1.7.0_85 +1 javac 14m 50s the patch passed -1 checkstyle 0m 26s Patch generated 1 new checkstyle issues in hadoop-common-project/hadoop-common (total was 45, now 42). +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 27s the patch passed +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_85 -1 unit 14m 14s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 14m 56s hadoop-common in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 36s Patch does not generate ASF License warnings. 130m 14s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.find.TestAnd   hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.ipc.TestDecayRpcScheduler   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.test.TestTimedOutTestsListener   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestName   hadoop.fs.shell.find.TestFind   hadoop.ipc.TestRPCWaitForProxy JDK v1.7.0_85 Failed junit tests hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.ha.TestZKFailoverController   hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestFind   hadoop.ipc.TestRPCWaitForProxy   hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774577/HDFS-8836-06.patch JIRA Issue HDFS-8836 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d5f994e55d78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f256d1 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13680/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13680/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13680/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Updated patch for the review comments. Akira Ajisaka, please review the changes.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Updated patch for the review comments. Akira Ajisaka , please review the changes.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Thanks for the review Akira Ajisaka. Sorry for late response, I will update the patch in sometime this week.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Thanks for the review Akira Ajisaka . Sorry for late response, I will update the patch in sometime this week.
        Hide
        jfilipiak Jan Filipiak added a comment -

        sorry for that many typos

        Show
        jfilipiak Jan Filipiak added a comment - sorry for that many typos
        Hide
        jfilipiak Jan Filipiak added a comment -

        Akira Ajisaka thank yu for rethinking this issue, the comments to the path you suggest make sense. I was always tempted todo something along the lines of:

        delimiter = cf.getOpt("skip-empty-file") ? "" : "\n";
        

        But to I dont have very strong opinions about how this should look in the end.

        Show
        jfilipiak Jan Filipiak added a comment - Akira Ajisaka thank yu for rethinking this issue, the comments to the path you suggest make sense. I was always tempted todo something along the lines of: delimiter = cf.getOpt( "skip-empty-file" ) ? "" : " \n"; But to I dont have very strong opinions about how this should look in the end.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Some comments from me.

        +        skipEmptyFileDelimiter = cf.getOpt("skip-empty-file") ? true : false;
        

        1. ? true : false is redundant, can be removed.

                  if (skipEmptyFileDelimiter && src.stat.getLen() == 0) {
                    continue;
                  }
                  FSDataInputStream in = src.fs.open(src.path);
                  try {
                    IOUtils.copyBytes(in, out, getConf(), false);
                    if (delimiter != null) {
                      out.write(delimiter.getBytes("UTF-8"));
                    }
                  } finally {
                    in.close();
                  }
        

        2. Can we skip opening empty file if the file length is zero as follows?

            if (src.stat.getLen() != 0) {
              try (FSDataInputStream in = src.fs.open(src.path)) {
                IOUtils.copyBytes(in, out, getConf(), false);
                writeDelimiter(out);
              }
            } else if (!skipEmptyFileDelimiter) {
              writeDelimiter(out);
            }
        
        private void writeDelimiter(FSDataOutputStream out) {
          ...
        }  
        
        TestFsShellCopy#testCopyMerge
            // directory with 3 files, should skip subdir
        

        3. An empty file is added, so there are 4 files.

        Show
        ajisakaa Akira Ajisaka added a comment - Some comments from me. + skipEmptyFileDelimiter = cf.getOpt( "skip-empty-file" ) ? true : false ; 1. ? true : false is redundant, can be removed. if (skipEmptyFileDelimiter && src.stat.getLen() == 0) { continue ; } FSDataInputStream in = src.fs.open(src.path); try { IOUtils.copyBytes(in, out, getConf(), false ); if (delimiter != null ) { out.write(delimiter.getBytes( "UTF-8" )); } } finally { in.close(); } 2. Can we skip opening empty file if the file length is zero as follows? if (src.stat.getLen() != 0) { try (FSDataInputStream in = src.fs.open(src.path)) { IOUtils.copyBytes(in, out, getConf(), false ); writeDelimiter(out); } } else if (!skipEmptyFileDelimiter) { writeDelimiter(out); } private void writeDelimiter(FSDataOutputStream out) { ... } TestFsShellCopy#testCopyMerge // directory with 3 files, should skip subdir 3. An empty file is added, so there are 4 files.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Sorry for late response.

        One could set up many oozie coordinators that would wait for A/_SUCCESS and then start processing it. There would be no safe time to delete the file as one is always in danger of having one of the cooridnators not executed as they didn't find its "dataset" file.

        Reasonable for me. I'll review your patch.

        Show
        ajisakaa Akira Ajisaka added a comment - Sorry for late response. One could set up many oozie coordinators that would wait for A/_SUCCESS and then start processing it. There would be no safe time to delete the file as one is always in danger of having one of the cooridnators not executed as they didn't find its "dataset" file. Reasonable for me. I'll review your patch.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Hi Akira Ajisaka I think it is reasonable to provide this option based on Jan Filipiak use case when multiple applications are referring and one of application is trying to make a collective result by referring output.

        Do you agree ?

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Hi Akira Ajisaka I think it is reasonable to provide this option based on Jan Filipiak use case when multiple applications are referring and one of application is trying to make a collective result by referring output. Do you agree ?
        Hide
        jfilipiak Jan Filipiak added a comment -

        Akira Ajisaka
        Your approach is quite similliar to the one followed in the ticket. Find zero size files and treat them differently.
        Ideally I would like skipping the empty files from the moment they get created, but this is 1) unpractical as many different applications show the behavior of creating empty files and all of them had to be fixed and 2) sometimes these emtpy files are required for some purposes and only harmful during the getmerge step. To explain case 2 a little bit more, imagine an application that uses directory A as an intermediate output that gets used by many other applications. Sqoop makes a good example for this. One could set up many oozie coordinators that would wait for A/_SUCCESS and then start processing it. There would be no safe time to delete the file as one is always in danger of having one of the cooridnators not executed as they didn't find its "dataset" file.

        Those two are the main reasons I consider this patch very helpfull. If namespacesize gets a problem one can always start tackling this at a different level. Applying the default Hiddenfilefilter would help in my case, but this would need a option aswell and just skipping all the empty files is semantically more correct in this case.

        Show
        jfilipiak Jan Filipiak added a comment - Akira Ajisaka Your approach is quite similliar to the one followed in the ticket. Find zero size files and treat them differently. Ideally I would like skipping the empty files from the moment they get created, but this is 1) unpractical as many different applications show the behavior of creating empty files and all of them had to be fixed and 2) sometimes these emtpy files are required for some purposes and only harmful during the getmerge step. To explain case 2 a little bit more, imagine an application that uses directory A as an intermediate output that gets used by many other applications. Sqoop makes a good example for this. One could set up many oozie coordinators that would wait for A/_SUCCESS and then start processing it. There would be no safe time to delete the file as one is always in danger of having one of the cooridnators not executed as they didn't find its "dataset" file. Those two are the main reasons I consider this patch very helpfull. If namespacesize gets a problem one can always start tackling this at a different level. Applying the default Hiddenfilefilter would help in my case, but this would need a option aswell and just skipping all the empty files is semantically more correct in this case.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        I'm thinking the new option is not reasonable. For HDFS, empty files should be deleted because empty files need no extra storage but extra NN heap. Therefore it's better to remove empty files before merging them.
        Even though you cannot delete empty files for some reasons, I prefer using find command such as "hadoop fs -find <dir> -type f AND -depth 1 AND (NOT -size 0) | xargs hadoop fs -getmerge -nl" rather than adding the new option. Unfortunately, find command for HDFS is still in development, so we cannot use -type, -depth, or -size option.

        Show
        ajisakaa Akira Ajisaka added a comment - I'm thinking the new option is not reasonable. For HDFS, empty files should be deleted because empty files need no extra storage but extra NN heap. Therefore it's better to remove empty files before merging them. Even though you cannot delete empty files for some reasons, I prefer using find command such as "hadoop fs -find <dir> -type f AND -depth 1 AND (NOT -size 0) | xargs hadoop fs -getmerge -nl" rather than adding the new option. Unfortunately, find command for HDFS is still in development, so we cannot use -type, -depth, or -size option.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        checkstyle issue can be ignored. whitespace issue, will fix along with other comments on patch from reviewer

        Show
        kanaka Kanaka Kumar Avvaru added a comment - checkstyle issue can be ignored. whitespace issue, will fix along with other comments on patch from reviewer
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 48s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 6s There were no new javac warning messages.
        +1 javadoc 10m 20s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 8s Site still builds.
        -1 checkstyle 1m 9s The applied patch generated 1 new checkstyle issues (total was 44, now 41).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 54s Tests passed in hadoop-common.
            71m 56s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12756238/HDFS-8836-05.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / bf2f2b4
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12479/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12479/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 20s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 site 3m 8s Site still builds. -1 checkstyle 1m 9s The applied patch generated 1 new checkstyle issues (total was 44, now 41). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 54s Tests passed in hadoop-common.     71m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756238/HDFS-8836-05.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / bf2f2b4 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12479/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12479/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12479/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        rebased the patch on latest trunk code

        Show
        kanaka Kanaka Kumar Avvaru added a comment - rebased the patch on latest trunk code
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Akira Ajisaka, Can you please review the patch.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Akira Ajisaka , Can you please review the patch.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        checkstyle issue can be ignored. whitespace issue, will fix later based on any other comments on patch

        Show
        kanaka Kanaka Kumar Avvaru added a comment - checkstyle issue can be ignored. whitespace issue, will fix later based on any other comments on patch
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 6s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 47s There were no new javac warning messages.
        +1 javadoc 10m 6s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 1s Site still builds.
        -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 45, now 42).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 17s Tests passed in hadoop-common.
            69m 45s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753929/HDFS-8836-04.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / 09c64ba
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12274/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12274/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 6s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 site 3m 1s Site still builds. -1 checkstyle 1m 11s The applied patch generated 1 new checkstyle issues (total was 45, now 42). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 17s Tests passed in hadoop-common.     69m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753929/HDFS-8836-04.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / 09c64ba checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12274/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12274/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12274/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Re-based patch for trunk changes in FileSystemShell.md

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Re-based patch for trunk changes in FileSystemShell.md
        Hide
        jfilipiak Jan Filipiak added a comment -

        Thanks for taking this into consideration Kanaka Kumar Avvaru, one could probably think about skipping the open + the readFully call in the zero length case. OTOH that is probably a rare case and I dont think one needs to really pay attention. Looking forward to use this feature in an official release. Thanks

        Show
        jfilipiak Jan Filipiak added a comment - Thanks for taking this into consideration Kanaka Kumar Avvaru , one could probably think about skipping the open + the readFully call in the zero length case. OTOH that is probably a rare case and I dont think one needs to really pay attention. Looking forward to use this feature in an official release. Thanks
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 30s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 54s There were no new javac warning messages.
        +1 javadoc 10m 17s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 18s Site still builds.
        -1 checkstyle 1m 9s The applied patch generated 3 new checkstyle issues (total was 45, now 47).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 7s Tests passed in hadoop-common.
            70m 45s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753354/HDFS-8836-03.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / caa04de
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12217/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12217/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12217/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12217/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 30s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 10m 17s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 18s Site still builds. -1 checkstyle 1m 9s The applied patch generated 3 new checkstyle issues (total was 45, now 47). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 7s Tests passed in hadoop-common.     70m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753354/HDFS-8836-03.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / caa04de checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12217/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12217/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12217/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12217/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Updated patch for above review comment and check style issue for indentation

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Updated patch for above review comment and check style issue for indentation
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Thanks Jan Filipiak I missed to observe stats in PathData, yes the extra call is not required. I will update the patch.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Thanks Jan Filipiak I missed to observe stats in PathData , yes the extra call is not required. I will update the patch.
        Hide
        jfilipiak Jan Filipiak added a comment -

        Hi Kanaka Kumar Avvaru

        thank you for looking into this and your patch. I was just wondering if the call src.fs.getFileStatus is really necessary, the comment above processPath and processPath itself indicate that the FileStatus is already set for the PathData objects in the srcs list. It looks like this extra roundtrip to the NN might be skipped.

        Show
        jfilipiak Jan Filipiak added a comment - Hi Kanaka Kumar Avvaru thank you for looking into this and your patch. I was just wondering if the call src.fs.getFileStatus is really necessary, the comment above processPath and processPath itself indicate that the FileStatus is already set for the PathData objects in the srcs list. It looks like this extra roundtrip to the NN might be skipped.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 22m 10s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 23s There were no new javac warning messages.
        +1 javadoc 10m 52s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 6s Site still builds.
        -1 checkstyle 1m 10s The applied patch generated 2 new checkstyle issues (total was 45, now 46).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 6s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 23m 8s Tests failed in hadoop-common.
            73m 40s  



        Reason Tests
        Failed unit tests hadoop.fs.sftp.TestSFTPFileSystem



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752298/HDFS-8836-02.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / eee0d45
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12123/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12123/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 22m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 23s There were no new javac warning messages. +1 javadoc 10m 52s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 site 3m 6s Site still builds. -1 checkstyle 1m 10s The applied patch generated 2 new checkstyle issues (total was 45, now 46). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 6s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 23m 8s Tests failed in hadoop-common.     73m 40s   Reason Tests Failed unit tests hadoop.fs.sftp.TestSFTPFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752298/HDFS-8836-02.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / eee0d45 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12123/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12123/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Updated patch for jenkin reported errors

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Updated patch for jenkin reported errors
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 3s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 50s There were no new javac warning messages.
        +1 javadoc 9m 58s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 0s Site still builds.
        -1 checkstyle 1m 8s The applied patch generated 2 new checkstyle issues (total was 45, now 47).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 16s Tests failed in hadoop-common.
            68m 30s  



        Reason Tests
        Failed unit tests hadoop.cli.TestCLI



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752203/HDFS-8836-01.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / eee0d45
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12114/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12114/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 3s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 58s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. -1 checkstyle 1m 8s The applied patch generated 2 new checkstyle issues (total was 45, now 47). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 16s Tests failed in hadoop-common.     68m 30s   Reason Tests Failed unit tests hadoop.cli.TestCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752203/HDFS-8836-01.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / eee0d45 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12114/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12114/console This message was automatically generated.
        Hide
        kanaka Kanaka Kumar Avvaru added a comment -

        Attached a patch to add a new flag -skip-empty-file and check the file length before copy content.

        Show
        kanaka Kanaka Kumar Avvaru added a comment - Attached a patch to add a new flag -skip-empty-file and check the file length before copy content.

          People

          • Assignee:
            kanaka Kanaka Kumar Avvaru
            Reporter:
            jfilipiak Jan Filipiak
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development