Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3520

Add transfer rate logging to TransferFsImage

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Logging the transfer rate for images and edits in TransferFsImage is useful for debugging network issues, especially when using dfs.datanode.balance.bandwidthPerSec.

      1. hdfs-3520-b1.txt
        1 kB
        Eli Collins
      2. hdfs-3520.txt
        1 kB
        Eli Collins
      3. hdfs-3520.txt
        2 kB
        Eli Collins
      4. hdfs-3520.txt
        2 kB
        Eli Collins
      5. hdfs-3520.txt
        2 kB
        Eli Collins
      6. hdfs-3520.txt
        2 kB
        Eli Collins

        Activity

        Eli Collins created issue -
        Hide
        Eli Collins added a comment -

        Patch for branch-1.

        Show
        Eli Collins added a comment - Patch for branch-1.
        Eli Collins made changes -
        Field Original Value New Value
        Attachment hdfs-3520.txt [ 12531481 ]
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531481 ]
        Eli Collins made changes -
        Attachment hdfs-3520-b1.txt [ 12531482 ]
        Hide
        Eli Collins added a comment -

        Patch for trunk.

        Show
        Eli Collins added a comment - Patch for trunk.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531483 ]
        Eli Collins made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Todd Lipcon added a comment -

        Looks good, except I'd use "Util.monotonicNow()" so that the result isn't thrown off if the clock changes. Also probably want to round the resulting value to nearest int, no?

        Show
        Todd Lipcon added a comment - Looks good, except I'd use "Util.monotonicNow()" so that the result isn't thrown off if the clock changes. Also probably want to round the resulting value to nearest int, no?
        Hide
        Eli Collins added a comment -

        That's better, updated patch. Also using String.format to make the output prettier.

        Show
        Eli Collins added a comment - That's better, updated patch. Also using String.format to make the output prettier.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531491 ]
        Hide
        Todd Lipcon added a comment -

        Am afraid of a div-by-zero here as well... unlikely to transfer in <1ms, but not impossible for a freshly formatted image, right? Perhaps should add Math.max(duration, 1) there to avoid a potential exception thrown. While you're at it, want to make the number of seconds a float as well?

        Show
        Todd Lipcon added a comment - Am afraid of a div-by-zero here as well... unlikely to transfer in <1ms, but not impossible for a freshly formatted image, right? Perhaps should add Math.max(duration, 1) there to avoid a potential exception thrown. While you're at it, want to make the number of seconds a float as well?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531491/hdfs-3520.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531491/hdfs-3520.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2628//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Good suggestion, updated patch.

        Show
        Eli Collins added a comment - Good suggestion, updated patch.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531693 ]
        Hide
        Todd Lipcon added a comment -

        Looks good to me. I might change the Math.max arg to 0.001 instead of 1.0 there, but feel free to address that on commit. +1

        Show
        Todd Lipcon added a comment - Looks good to me. I might change the Math.max arg to 0.001 instead of 1.0 there, but feel free to address that on commit. +1
        Hide
        Eli Collins added a comment -

        Updated patch to fix the findbugs warning.

        Show
        Eli Collins added a comment - Updated patch to fix the findbugs warning.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531725 ]
        Hide
        Eli Collins added a comment -

        Right patch this time.

        Show
        Eli Collins added a comment - Right patch this time.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531727 ]
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531725 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531693/hdfs-3520.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531693/hdfs-3520.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2635//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        findbugs wants the division to be on it's own line. Verified this by running findbugs locally.

        Show
        Eli Collins added a comment - findbugs wants the division to be on it's own line. Verified this by running findbugs locally.
        Eli Collins made changes -
        Attachment hdfs-3520.txt [ 12531753 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531753/hdfs-3520.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2639//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2639//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531753/hdfs-3520.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2639//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2639//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        The test failure is unrelated, filed HDFS-3524 for it. This just adds a LOG.info so I the javadoc warning is unrelated as well.

        Show
        Eli Collins added a comment - The test failure is unrelated, filed HDFS-3524 for it. This just adds a LOG.info so I the javadoc warning is unrelated as well.
        Eli Collins made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Affects Version/s 1.2.0 [ 12321657 ]
        Target Version/s 1.2.0, 2.0.1-alpha [ 12321657, 12321440 ] 2.0.1-alpha [ 12321440 ]
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks for the review Todd.

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks for the review Todd.
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Target Version/s 2.0.1-alpha [ 12321440 ]
        Fix Version/s 2.0.1-alpha [ 12321440 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2367 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2367/)
        HDFS-3520. Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2367 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2367/ ) HDFS-3520 . Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2418/)
        HDFS-3520. Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2418/ ) HDFS-3520 . Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2345 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2345/)
        HDFS-3520. Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2345 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2345/ ) HDFS-3520 . Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/)
        HDFS-3520. Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/ ) HDFS-3520 . Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/)
        HDFS-3520. Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/ ) HDFS-3520 . Add transfer rate logging to TransferFsImage. Contributed by Eli Collins (Revision 1349117) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349117 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
        Eli Collins made changes -
        Assignee Eli Collins [ eli2 ] Eli Collins [ eli ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322472 ]
        Fix Version/s 2.1.0-alpha [ 12321440 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Suresh Srinivas made changes -
        Priority Major [ 3 ] Minor [ 4 ]

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development