Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4569

Small image transfer related cleanups.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The initial patch in HDFS-1490 has a couple small errors. It missed adding the new configuration key "dfs.image.transfer.timeout" to the hdfs-default.xml, and kept an explanatory comment from an earlier version of the patch that is no longer correct. Also, the default timeout of 1 minute is too short and can be increased.

      1. hdfs-4569-4.patch
        3 kB
        Andrew Wang
      2. hdfs-4569-3.patch
        2 kB
        Andrew Wang
      3. hdfs-4569-2.patch
        2 kB
        Andrew Wang
      4. hdfs-4569-1.patch
        2 kB
        Andrew Wang

        Activity

        Hide
        Andrew Wang added a comment -

        Patch attached. No code changes, so no tests.

        Show
        Andrew Wang added a comment - Patch attached. No code changes, so no tests.
        Hide
        Suresh Srinivas added a comment -

        Sorry I have not looked at the original jira. The image transfer should complete with in the time defined in "dfs.image.transfer.timeout"? If so can we explicitly describe that in the document you added?

        I know this is not related to this jira, but, is 60 seconds default sufficient for large clusters?

        Show
        Suresh Srinivas added a comment - Sorry I have not looked at the original jira. The image transfer should complete with in the time defined in "dfs.image.transfer.timeout"? If so can we explicitly describe that in the document you added? I know this is not related to this jira, but, is 60 seconds default sufficient for large clusters?
        Hide
        Andrew Wang added a comment -

        Hey Suresh,

        I tried making the text more explicit, let me know if it's still unclear.

        We've hit the default timeout with somewhat large fsimages, in the 2GB+ range. Using HTTPURLConnection has some timeout limitations, and the timeout and bandwidthPerSec configuration options are kind of poor since they depend on image size (see HDFS-4301).

        Show
        Andrew Wang added a comment - Hey Suresh, I tried making the text more explicit, let me know if it's still unclear. We've hit the default timeout with somewhat large fsimages, in the 2GB+ range. Using HTTPURLConnection has some timeout limitations, and the timeout and bandwidthPerSec configuration options are kind of poor since they depend on image size (see HDFS-4301 ).
        Hide
        Suresh Srinivas added a comment -

        We've hit the default timeout with somewhat large fsimages, in the 2GB+ range.

        I am surprised default of 1 minute was chosen. If a default of 10 minutes is chosen, it should have been okay for HDFS-1490, given it is trying to solve issues related to rare problems, right? Perhaps this can be done in another jira.

        A minor comment for the patch - just like "dfs.image.transfer.timeout" points to "dfs.image.transfer.bandwidthPerSec" in description, should there be a description in bandwidthPerSec to point to transfer.timeout?

        Show
        Suresh Srinivas added a comment - We've hit the default timeout with somewhat large fsimages, in the 2GB+ range. I am surprised default of 1 minute was chosen. If a default of 10 minutes is chosen, it should have been okay for HDFS-1490 , given it is trying to solve issues related to rare problems, right? Perhaps this can be done in another jira. A minor comment for the patch - just like "dfs.image.transfer.timeout" points to "dfs.image.transfer.bandwidthPerSec" in description, should there be a description in bandwidthPerSec to point to transfer.timeout?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572585/hdfs-4569-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4049//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4049//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572585/hdfs-4569-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4049//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4049//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572603/hdfs-4569-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4050//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4050//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572603/hdfs-4569-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4050//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4050//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Thanks for all the reviews. Another rev of the patch attached.

        I also filed HDFS-4570 for increasing the default timeout value.

        Show
        Andrew Wang added a comment - Thanks for all the reviews. Another rev of the patch attached. I also filed HDFS-4570 for increasing the default timeout value.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch. I will commit it shortly.

        Show
        Suresh Srinivas added a comment - +1 for the patch. I will commit it shortly.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > I also filed HDFS-4570 for increasing the default timeout value.

        You may not need to file HDFS-4570 since both changes of this and HDFS-4570 are quite minor. Why don't you combine them?

        Show
        Tsz Wo Nicholas Sze added a comment - > I also filed HDFS-4570 for increasing the default timeout value. You may not need to file HDFS-4570 since both changes of this and HDFS-4570 are quite minor. Why don't you combine them?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572624/hdfs-4569-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4051//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4051//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572624/hdfs-4569-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4051//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4051//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Sure Nicholas, I added that change to this newest patch.

        Show
        Andrew Wang added a comment - Sure Nicholas, I added that change to this newest patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12572645/hdfs-4569-4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4055//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4055//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572645/hdfs-4569-4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4055//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4055//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch.

        Show
        Suresh Srinivas added a comment - +1 for the patch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3439 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3439/)
        HDFS-4569. Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3439 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3439/ ) HDFS-4569 . Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Suresh Srinivas added a comment -

        I committed the change to trunk and branch-2. Thank you Andrew.

        Show
        Suresh Srinivas added a comment - I committed the change to trunk and branch-2. Thank you Andrew.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/)
        HDFS-4569. Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/ ) HDFS-4569 . Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/)
        HDFS-4569. Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/ ) HDFS-4569 . Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/)
        HDFS-4569. Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/ ) HDFS-4569 . Small image transfer related cleanups. Contributed by Andrew Wang. (Revision 1454233) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

          People

          • Assignee:
            Andrew Wang
            Reporter:
            Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development