Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11914

Add more diagnosis info for fsimage transfer failure.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Hit a fsimage download problem:

      Client tries to download fsimage, and got:

      WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: File http://x.y.z:50070/imagetransfer?getimage=1&txid=latest received length xyz is not of the advertised size abc.

      Basically client does not get enough fsimage data and finished prematurely without any exception thrown, until it finds the size of data received is smaller than expected. The client then closed the conenction to NN, that caused NN to report

      INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Connection closed by client

      This jira is to add some more information in logs to help debugging the sitaution. Specifically, report the stack trace when the connection is closed. And how much data has been sent at that point. etc.

      1. HDFS-11914.001.patch
        3 kB
        Yongjun Zhang
      2. HDFS-11914.002.patch
        3 kB
        Yongjun Zhang
      3. HDFS-11914.003.patch
        3 kB
        Yongjun Zhang
      4. HDFS-11914.004.patch
        4 kB
        Yongjun Zhang
      5. HDFS-11914.branch-2.001.patch
        3 kB
        Yongjun Zhang

        Activity

        Hide
        yzhangal Yongjun Zhang added a comment -

        Upload patch rev 001.

        Show
        yzhangal Yongjun Zhang added a comment - Upload patch rev 001.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 19s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 38s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        +1 checkstyle 0m 32s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 63m 13s hadoop-hdfs in the patch failed.
        -1 asflicense 0m 19s The patch generated 2 ASF License warnings.
        88m 52s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
          hadoop.fs.viewfs.TestViewFileSystemHdfs
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11914
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870933/HDFS-11914.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6b582713e0b6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8d9084e
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19743/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19743/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19743/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19743/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 19s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 63m 13s hadoop-hdfs in the patch failed. -1 asflicense 0m 19s The patch generated 2 ASF License warnings. 88m 52s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11914 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870933/HDFS-11914.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b582713e0b6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d9084e Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19743/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19743/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19743/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19743/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Uploaded rev 002 to include some more info.

        The failed tests are not related, running them locally succeeded.

        Show
        yzhangal Yongjun Zhang added a comment - Uploaded rev 002 to include some more info. The failed tests are not related, running them locally succeeded.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 22s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 33s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 38s the patch passed
        +1 unit 87m 12s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        113m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11914
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870995/HDFS-11914.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1d0eaea805cc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 056cc72
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19751/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19751/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 22s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 38s the patch passed +1 unit 87m 12s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 113m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11914 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870995/HDFS-11914.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d0eaea805cc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 056cc72 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19751/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19751/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Yongjun Zhang
        I reviewed the patch and mostly good. I suggest improve the readability of the log a little bit:
        in copyFileToStream:
        "Connection closed by client. Sent total=". add " bytes." at the end.
        " Size of last segment possibly sent=" can be rephrased as " Size of last segment intended to send="

        Could you also explain why use String.valueOf() to print fsImageName, a String?

        Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Yongjun Zhang I reviewed the patch and mostly good. I suggest improve the readability of the log a little bit: in copyFileToStream : "Connection closed by client. Sent total=". add " bytes." at the end. " Size of last segment possibly sent=" can be rephrased as " Size of last segment intended to send=" Could you also explain why use String.valueOf() to print fsImageName, a String? Thanks!
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Wei-Chiu Chuang, good comments.

        I'm uploading a patch that address the first two comments, About why using String.valueOf(), it's because I saw some code in the same method check if the fsImageName is null, which means there is a chance the fsimageName is null. I just want to be safe here.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Wei-Chiu Chuang , good comments. I'm uploading a patch that address the first two comments, About why using String.valueOf(), it's because I saw some code in the same method check if the fsImageName is null, which means there is a chance the fsimageName is null. I just want to be safe here.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        About why using String.valueOf(), it's because I saw some code in the same method check if the fsImageName is null, which means there is a chance the fsimageName is null.

        That shouldn't be necessary. The following code:

              String x = null;
              throw new IOException("abc" + x + "def");
        

        Gives me
        java.io.IOException: abcnulldef
        It doesn't cause NPE.

        Show
        jojochuang Wei-Chiu Chuang added a comment - About why using String.valueOf(), it's because I saw some code in the same method check if the fsImageName is null, which means there is a chance the fsimageName is null. That shouldn't be necessary. The following code: String x = null ; throw new IOException( "abc" + x + "def" ); Gives me java.io.IOException: abcnulldef It doesn't cause NPE.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 22m 32s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 15m 1s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 40s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 45s trunk passed
        +1 mvninstall 0m 56s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 89m 23s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        140m 5s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11914
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871020/HDFS-11914.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a71e1a5007c2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 73ecb19
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19752/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19752/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19752/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 22m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 1s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 56s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 89m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 140m 5s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11914 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871020/HDFS-11914.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a71e1a5007c2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 73ecb19 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19752/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19752/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19752/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Wei-Chiu Chuang.

        Good to know. Rev4 address your comments, and some further improvement to always report a message at server side about what has been sent to the client.

        Would you please take a look again? thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Wei-Chiu Chuang . Good to know. Rev4 address your comments, and some further improvement to always report a message at server side about what has been sent to the client. Would you please take a look again? thanks.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        LGTM +1. Thanks

        Show
        jojochuang Wei-Chiu Chuang added a comment - LGTM +1. Thanks
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 34s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 88m 16s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        113m 56s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11914
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871269/HDFS-11914.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f0f31c3db0a9 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / abdd609
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19778/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19778/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19778/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19778/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 34s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 52s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 88m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 113m 56s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11914 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871269/HDFS-11914.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f0f31c3db0a9 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / abdd609 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19778/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19778/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19778/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19778/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11827 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11827/)
        HDFS-11914. Add more diagnosis info for fsimage transfer failure. (yzhang: rev 6a28a2b914664fbbb5267f13e82d42b3bb49610c)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11827 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11827/ ) HDFS-11914 . Add more diagnosis info for fsimage transfer failure. (yzhang: rev 6a28a2b914664fbbb5267f13e82d42b3bb49610c) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks for the review Wei-Chiu Chuang, I committed to trunk and uploaded a branch-2 version. Would you please take a look? thanks,

        Show
        yzhangal Yongjun Zhang added a comment - Thanks for the review Wei-Chiu Chuang , I committed to trunk and uploaded a branch-2 version. Would you please take a look? thanks,
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 17m 52s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 48s branch-2 passed
        +1 compile 0m 36s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 42s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 24s branch-2 passed
        +1 mvnsite 0m 49s branch-2 passed
        +1 findbugs 2m 5s branch-2 passed
        +1 javadoc 0m 36s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 57s branch-2 passed with JDK v1.7.0_131
        +1 mvninstall 0m 43s the patch passed
        +1 compile 0m 35s the patch passed with JDK v1.8.0_131
        +1 javac 0m 35s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_131
        +1 javac 0m 39s the patch passed
        -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14)
        +1 mvnsite 0m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 8s the patch passed
        +1 javadoc 0m 34s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 55s the patch passed with JDK v1.7.0_131
        -1 unit 54m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        149m 52s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
        JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
          hadoop.hdfs.web.TestWebHdfsTimeouts
        JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HDFS-11914
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871380/HDFS-11914.branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 30813dcce0e4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 95c2978
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19782/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19782/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19782/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19782/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 52s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 48s branch-2 passed +1 compile 0m 36s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 42s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 24s branch-2 passed +1 mvnsite 0m 49s branch-2 passed +1 findbugs 2m 5s branch-2 passed +1 javadoc 0m 36s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 57s branch-2 passed with JDK v1.7.0_131 +1 mvninstall 0m 43s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_131 +1 javac 0m 35s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_131 +1 javac 0m 39s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) +1 mvnsite 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 0m 34s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 55s the patch passed with JDK v1.7.0_131 -1 unit 54m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 149m 52s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain   hadoop.hdfs.web.TestWebHdfsTimeouts JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HDFS-11914 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871380/HDFS-11914.branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 30813dcce0e4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 95c2978 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19782/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19782/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19782/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19782/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        +1 for the branch-2 patch. Could you please update the patch to address the trivial checkstyle issue when you commit the patch?

        Show
        jojochuang Wei-Chiu Chuang added a comment - +1 for the branch-2 patch. Could you please update the patch to address the trivial checkstyle issue when you commit the patch?
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Wei-Chiu Chuang much for the review. Sorry I missed the checkstyle issue before committing to trunk. Resolved it and committed to branch-2.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Wei-Chiu Chuang much for the review. Sorry I missed the checkstyle issue before committing to trunk. Resolved it and committed to branch-2.

          People

          • Assignee:
            yzhangal Yongjun Zhang
            Reporter:
            yzhangal Yongjun Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development