Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3170

Add more useful metrics for write latency

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the only write-latency related metric we expose is the total amount of time taken by opWriteBlock. This is practically useless, since (a) different blocks may be wildly different sizes, and (b) if the writer is only generating data slowly, it will make a block write take longer by no fault of the DN. I would like to propose two new metrics:
      1) flush-to-disk time: count how long it takes for each call to flush an incoming packet to disk (including the checksums). In most cases this will be close to 0, as it only flushes to buffer cache, but if the backing block device enters congested writeback, it can take much longer, which provides an interesting metric.
      2) round trip to downstream pipeline node: track the round trip latency for the part of the pipeline between the local node and its downstream neighbors. When we add a new packet to the ack queue, save the current timestamp. When we receive an ack, update the metric based on how long since we sent the original packet. This gives a metric of the total RTT through the pipeline. If we also include this metric in the ack to upstream, we can subtract the amount of time due to the later stages in the pipeline and have an accurate count of this particular link.

      1. hdfs-3170.txt
        16 kB
        Matthew Jacobs
      2. hdfs-3170.txt
        14 kB
        Matthew Jacobs
      3. hdfs-3170.txt
        12 kB
        Matthew Jacobs

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Another improvement would be to track the flush-to-disk time separately per data directory. This can help detect disks that are starting to go bad.

          Show
          Todd Lipcon added a comment - Another improvement would be to track the flush-to-disk time separately per data directory. This can help detect disks that are starting to go bad.
          Hide
          Matthew Jacobs added a comment -

          I've started working on this. I've implemented both metrics but I think I should write a test for the round trip to downstream pipeline metric.

          Show
          Matthew Jacobs added a comment - I've started working on this. I've implemented both metrics but I think I should write a test for the round trip to downstream pipeline metric.
          Hide
          Matthew Jacobs added a comment -

          The attached patch adds the write-latency related metrics described in this JIRA. The tests verify that the metrics are added. I manually checked that the averaged latency values were reasonable. For example, I added a sleep before taking the ack end time and then verified that the resulting metric (via jmx) was greater than the sleep time.

          Show
          Matthew Jacobs added a comment - The attached patch adds the write-latency related metrics described in this JIRA. The tests verify that the metrics are added. I manually checked that the averaged latency values were reasonable. For example, I added a sleep before taking the ack end time and then verified that the resulting metric (via jmx) was greater than the sleep time.
          Hide
          Matthew Jacobs added a comment -

          The attached patch adds the write-latency related metrics described in this JIRA. The tests verify that the metrics are added. I manually checked that the averaged latency values were reasonable. For example, I added a sleep before taking the ack end time and then verified that the resulting metric (via jmx) was greater than the sleep time.

          Show
          Matthew Jacobs added a comment - The attached patch adds the write-latency related metrics described in this JIRA. The tests verify that the metrics are added. I manually checked that the averaged latency values were reasonable. For example, I added a sleep before taking the ack end time and then verified that the resulting metric (via jmx) was greater than the sleep time.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533205/hdfs-3170.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2696//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2696//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533205/hdfs-3170.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2696//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2696//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Looks pretty good. A few comments:

          • Can you use System.nanoTime so that it's (a) a monotonic clock instead of time-of-day, and (b) higher granularity than MS? I think a lot of these metrics will end up sub-millisecond.
          • In flushOrSync() I think you can avoid having quite so many calls to get the time – eg after flush(), the time there is the same as the start time of sync(). They're not super expensive, but they are syscalls, so let's be efficient and call it the minimal number of times.
          • For all your variables/metric names, can you please add the units? eg "packetAckRoundTripTimeNanos"?
          • I'm not sure the math's quite right: in the Ack itself, we should be including the total ack time from the whole downstream pipeline, not just the immediate next hop. In ASCII form:
          .
                    A->         B->        C->        D->
          Client          DN1        DN2        DN3        DN4
                    <-H         <-G        <-F        <-E
          

          G's downstream RTT time should = (F.recvTime - C.sendTime), not (F.recvTime - C.sendTime - (E.recvTime - D.sendTime)). Does that make sense? Otherwise DN1 will calculate the wrong metric.

          Show
          Todd Lipcon added a comment - Looks pretty good. A few comments: Can you use System.nanoTime so that it's (a) a monotonic clock instead of time-of-day, and (b) higher granularity than MS? I think a lot of these metrics will end up sub-millisecond. In flushOrSync() I think you can avoid having quite so many calls to get the time – eg after flush(), the time there is the same as the start time of sync(). They're not super expensive, but they are syscalls, so let's be efficient and call it the minimal number of times. For all your variables/metric names, can you please add the units? eg "packetAckRoundTripTimeNanos"? I'm not sure the math's quite right: in the Ack itself, we should be including the total ack time from the whole downstream pipeline, not just the immediate next hop. In ASCII form: . A-> B-> C-> D-> Client DN1 DN2 DN3 DN4 <-H <-G <-F <-E G's downstream RTT time should = (F.recvTime - C.sendTime), not (F.recvTime - C.sendTime - (E.recvTime - D.sendTime)). Does that make sense? Otherwise DN1 will calculate the wrong metric.
          Hide
          Andy Isaacson added a comment -

          Can you use System.nanoTime so that it's (a) a monotonic clock instead of time-of-day, and (b) higher granularity than MS? I think a lot of these metrics will end up sub-millisecond.

          You may want to review hdfs/server/common/Util.java:monotonicNow, although it returns milliseconds rather than ns.

          They're not super expensive, but they are syscalls, so let's be efficient and call it the minimal number of times.

          clock_gettime is a vsyscall, so it's really pretty cheap. Looks like about 60 ns or 180 clock cycles for CLOCK_MONOTONIC (tested on Xeon X5670, 3GHz). (Interestingly CLOCK_MONOTONIC_RAW which I expected to be faster by skipping the adjustment, is actually slower at 240 ns on the same Xeon.)

          Show
          Andy Isaacson added a comment - Can you use System.nanoTime so that it's (a) a monotonic clock instead of time-of-day, and (b) higher granularity than MS? I think a lot of these metrics will end up sub-millisecond. You may want to review hdfs/server/common/Util.java:monotonicNow, although it returns milliseconds rather than ns. They're not super expensive, but they are syscalls, so let's be efficient and call it the minimal number of times. clock_gettime is a vsyscall, so it's really pretty cheap. Looks like about 60 ns or 180 clock cycles for CLOCK_MONOTONIC (tested on Xeon X5670, 3GHz). (Interestingly CLOCK_MONOTONIC_RAW which I expected to be faster by skipping the adjustment, is actually slower at 240 ns on the same Xeon.)
          Hide
          Matthew Jacobs added a comment -

          Thanks, Todd and Andy.

          I agree about the timing issues, I'll use System.nanoTime() rather than milliseconds since we're interested in capturing sub-millisecond latency. The reason I didn't use nanoTime from the beginning was to be consistent with the fsync metric, though now I'm thinking it might also be better to report fsync in nanoseconds as well. And if I want to re-use the time after the flush() and before the sync(), I'll have to use the high resolution clock for all these timestamps, anyway. In this case, I'd rename fsync to fsyncNanos or something. If you think it would be better to keep fsync in milliseconds, I'll still use the monotonic clock and report the metrics in milliseconds.

          What do you think?

          You're right regarding the ack math- I worked through that incorrectly. Fortunately it's a really simple fix.

          I'll post an updated patch soon.

          Show
          Matthew Jacobs added a comment - Thanks, Todd and Andy. I agree about the timing issues, I'll use System.nanoTime() rather than milliseconds since we're interested in capturing sub-millisecond latency. The reason I didn't use nanoTime from the beginning was to be consistent with the fsync metric, though now I'm thinking it might also be better to report fsync in nanoseconds as well. And if I want to re-use the time after the flush() and before the sync(), I'll have to use the high resolution clock for all these timestamps, anyway. In this case, I'd rename fsync to fsyncNanos or something. If you think it would be better to keep fsync in milliseconds, I'll still use the monotonic clock and report the metrics in milliseconds. What do you think? You're right regarding the ack math- I worked through that incorrectly. Fortunately it's a really simple fix. I'll post an updated patch soon.
          Hide
          Matthew Jacobs added a comment -

          Updated patch with changes based on Todd's comments.

          I left the fsync metric as-is for now, but we can consider reporting the time in nanoseconds in the future.

          Show
          Matthew Jacobs added a comment - Updated patch with changes based on Todd's comments. I left the fsync metric as-is for now, but we can consider reporting the time in nanoseconds in the future.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12534171/hdfs-3170.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2728//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2728//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534171/hdfs-3170.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2728//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2728//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -
          +  public final static long NANOSECONDS_PER_MILLISECOND = 1000000;
          +
          

          Instead, use TimeUnit.convert?

          But actually, I think we should change the fsync metric to nanos or microseconds anyway. It's a pretty new one that hasn't been in any releases yet, so I think it's reasonable for us to change its units now for consistency.

          What do you think about making all of these metrics in microseconds? It seems a more natural unit for admins to understand, whereas nanos are way more precise than the timers we're using.


          +                      LOG.debug("Calculated invalid ack time: " + ackTimeNanos + "ns.");
          

          This should be guarded with a isDebugEnabled() check to prevent a possible perf issue if for some reason it gets triggered often.

          In the case it's triggered, shouldn't we add an RTT=0 to the metric?


          +    final long ackEnqueueTimeNanos;
          

          Rename to ackEnqueueNanoTime – otherwise it sounds like it's actually an elapsed time duration, rather than a timestamp. Or perhaps ackEnqueueTimestampNanos.

          Otherwise looking good!

          Show
          Todd Lipcon added a comment - + public final static long NANOSECONDS_PER_MILLISECOND = 1000000; + Instead, use TimeUnit.convert ? But actually, I think we should change the fsync metric to nanos or microseconds anyway. It's a pretty new one that hasn't been in any releases yet, so I think it's reasonable for us to change its units now for consistency. What do you think about making all of these metrics in microseconds? It seems a more natural unit for admins to understand, whereas nanos are way more precise than the timers we're using. + LOG.debug( "Calculated invalid ack time: " + ackTimeNanos + "ns." ); This should be guarded with a isDebugEnabled() check to prevent a possible perf issue if for some reason it gets triggered often. In the case it's triggered, shouldn't we add an RTT=0 to the metric? + final long ackEnqueueTimeNanos; Rename to ackEnqueueNanoTime – otherwise it sounds like it's actually an elapsed time duration, rather than a timestamp. Or perhaps ackEnqueueTimestampNanos . Otherwise looking good!
          Hide
          Matthew Jacobs added a comment -

          Thanks, Todd. I didn't know about TimeUnit.convert in Java, that's a better way to to the conversions.

          Regarding switching to microseconds, I don't really have a strong opinion, but I'm more used to seeing nanoseconds and milliseconds than microseconds in the context of timing, even though the high resolution timer resolution is likely only in microseconds anyway. Also, given that there are other metrics in nanoseconds, I think it would be better just to be consistent. I'm happy to change this if you'd prefer microseconds, though.

          In the case it's triggered, shouldn't we add an RTT=0 to the metric?

          My thought was that getting a negative number really shouldn't happen because the ack enqueue timestamp is taken before the actual packet send and the ack recv timestamp is taken after the packet is received, so getting a downstream ack duration must be strictly less than (or equal, though unlikely) to the (ackRecv-ackEnqueue) duration unless something is wrong, because logically it shouldn't happen. If something is wrong, reporting 0 seems like a guess to me and I'd rather not report anything but would prefer to just log the issue instead. I might be missing something though, what do you think?

          I'll change fsync and will rename ackEnqueueTimeNanos.

          Show
          Matthew Jacobs added a comment - Thanks, Todd. I didn't know about TimeUnit.convert in Java, that's a better way to to the conversions. Regarding switching to microseconds, I don't really have a strong opinion, but I'm more used to seeing nanoseconds and milliseconds than microseconds in the context of timing, even though the high resolution timer resolution is likely only in microseconds anyway. Also, given that there are other metrics in nanoseconds, I think it would be better just to be consistent. I'm happy to change this if you'd prefer microseconds, though. In the case it's triggered, shouldn't we add an RTT=0 to the metric? My thought was that getting a negative number really shouldn't happen because the ack enqueue timestamp is taken before the actual packet send and the ack recv timestamp is taken after the packet is received, so getting a downstream ack duration must be strictly less than (or equal, though unlikely) to the (ackRecv-ackEnqueue) duration unless something is wrong, because logically it shouldn't happen. If something is wrong, reporting 0 seems like a guess to me and I'd rather not report anything but would prefer to just log the issue instead. I might be missing something though, what do you think? I'll change fsync and will rename ackEnqueueTimeNanos.
          Hide
          Matthew Jacobs added a comment -

          Also, I realized that I didn't measure the flushes in BlockReceiver.close(), so I'm adding that now.

          Show
          Matthew Jacobs added a comment - Also, I realized that I didn't measure the flushes in BlockReceiver.close(), so I'm adding that now.
          Hide
          Matthew Jacobs added a comment -

          Updated the patch with a few small changes based on Todd's latest comments. Still open to feedback on the things we were discussing.

          Show
          Matthew Jacobs added a comment - Updated the patch with a few small changes based on Todd's latest comments. Still open to feedback on the things we were discussing.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535141/hdfs-3170.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHDFSTrash

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2740//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2740//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535141/hdfs-3170.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2740//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2740//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1, the patch looks good to me. Thanks for these nice new metrics, Matt.

          Show
          Todd Lipcon added a comment - +1, the patch looks good to me. Thanks for these nice new metrics, Matt.
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 and trunk. Thanks, Matt!

          Show
          Todd Lipcon added a comment - Committed to branch-2 and trunk. Thanks, Matt!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2427 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2427/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2427 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2427/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2495 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2495/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2495 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2495/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2445 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2445/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2445 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2445/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1095/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1095/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1128/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1128/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
          HDFS-3170. Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/ ) HDFS-3170 . Add more useful metrics for write latency. Contributed by Matthew Jacobs. (Revision 1357970) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1357970 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java

            People

            • Assignee:
              Matthew Jacobs
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development