Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 3.0.0-alpha4
-
Component/s: metrics
-
Labels:
-
Target Version/s:
-
Hadoop Flags:Incompatible change, Reviewed
-
Release Note:Remove the BlockReport(NumOps,AvgTime) metrics emitted under the NameNodeActivity context in favor of StorageBlockReport(NumOps,AvgTime) which more accurately represent the metric. Same for the corresponding quantile metrics.
Description
Currently the BlockReport(NumOps|AvgTime) metrics emitted under the RpcDetailedActivity context and those emitted under the NameNodeActivity context are actually reporting different things despite having the same name. NameNodeActivity reports the count/time of per storage block reports, whereas RpcDetailedActivity reports the count/time of per datanode block reports. This makes for a confusing experience with two metrics having the same name reporting different values.
We already have the StorageBlockReportsOps metric under NameNodeActivity. Can we make StorageBlockReport a MutableRate metric and remove NameNodeActivity#BlockReport metric? Open to other suggestions about how to address this as well. The 3.0 release seems a good time to make this incompatible change.
Issue Links
- relates to
-
HADOOP-14492
RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Can we make StorageBlockReport a MutableRate metric and remove NameNodeActivity#BlockReport metric
This sounds good to me (as a 3.0 change).
Pinging Andrew Wang for opinion on breaking compatibility in this case.
Sounds good to me.
Thanks both for looking, will post a patch soon.
Attaching initial patch. I updated the quantile metrics for BlockReports as well since they also reflect individual storages.
Old metrics output:
}, {
"name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010",
"BlockReportNumOps" : 2,
"BlockReportAvgTime" : 42.0,
...
}, {
"name" : "Hadoop:service=NameNode,name=NameNodeActivity",
"BlockReport60sNumOps" : 0,
"BlockReport60s50thPercentileLatency" : 0,
"BlockReport60s75thPercentileLatency" : 0,
"BlockReport60s90thPercentileLatency" : 0,
"BlockReport60s95thPercentileLatency" : 0,
"BlockReport60s99thPercentileLatency" : 0,
"StorageBlockReportsOps" : 4,
"BlockReportOps" : 4,
"BlockReportAvgTime" : 5.5,
...
}, {
New metrics output:
}, {
"name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010",
"BlockReportNumOps" : 2,
"BlockReportAvgTime" : 42.0,
...
}, {
"name" : "Hadoop:service=NameNode,name=NameNodeActivity",
"StorageBlockReport60sNumOps" : 0,
"StorageBlockReport60s50thPercentileLatency" : 0,
"StorageBlockReport60s75thPercentileLatency" : 0,
"StorageBlockReport60s90thPercentileLatency" : 0,
"StorageBlockReport60s95thPercentileLatency" : 0,
"StorageBlockReport60s99thPercentileLatency" : 0,
"StorageBlockReportNumOps" : 4,
"StorageBlockReportAvgTime" : 5.5,
...
}, {
I also would like to point out this is consistent with the DataNode BlockReport metric:
}, {
"name" : "Hadoop:service=DataNode,name=DataNodeActivity-127.0.0.1-58011",
"BlockReportsNumOps" : 2,
"BlockReportsAvgTime" : 65.0,
...
}, {
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 14s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 13m 20s | trunk passed |
| +1 | compile | 0m 58s | trunk passed |
| +1 | checkstyle | 0m 41s | trunk passed |
| +1 | mvnsite | 1m 1s | trunk passed |
| +1 | findbugs | 1m 51s | trunk passed |
| +1 | javadoc | 0m 44s | trunk passed |
| +1 | mvninstall | 0m 51s | the patch passed |
| +1 | compile | 0m 47s | the patch passed |
| +1 | javac | 0m 47s | the patch passed |
| -0 | checkstyle | 0m 36s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 242 unchanged - 4 fixed = 246 total (was 246) |
| +1 | mvnsite | 0m 54s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 48s | the patch passed |
| +1 | javadoc | 0m 38s | the patch passed |
| -1 | unit | 68m 3s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
| 94m 48s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12872094/HADOOP-14502.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux a54307184be9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5672ae7 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
v001 patch to fix unit test failure
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 20s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 13m 30s | trunk passed |
| +1 | compile | 0m 47s | trunk passed |
| +1 | checkstyle | 0m 39s | trunk passed |
| +1 | mvnsite | 0m 55s | trunk passed |
| +1 | findbugs | 1m 44s | trunk passed |
| +1 | javadoc | 0m 39s | trunk passed |
| +1 | mvninstall | 0m 49s | the patch passed |
| +1 | compile | 0m 45s | the patch passed |
| +1 | javac | 0m 45s | the patch passed |
| -0 | checkstyle | 0m 35s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 264 unchanged - 4 fixed = 268 total (was 268) |
| +1 | mvnsite | 0m 51s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 45s | the patch passed |
| +1 | javadoc | 0m 37s | the patch passed |
| -1 | unit | 87m 20s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 113m 27s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12872111/HADOOP-14502.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 78a9603618da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a062374 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
v002 fixes line length checkstyle... My mistake for not addressing in v2. The two checkstyle about private variables is standard for all of the metrics variables.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 19s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 14m 44s | trunk passed |
| +1 | compile | 0m 55s | trunk passed |
| +1 | checkstyle | 0m 45s | trunk passed |
| +1 | mvnsite | 1m 5s | trunk passed |
| +1 | findbugs | 1m 53s | trunk passed |
| +1 | javadoc | 0m 44s | trunk passed |
| +1 | mvninstall | 0m 48s | the patch passed |
| +1 | compile | 0m 57s | the patch passed |
| +1 | javac | 0m 57s | the patch passed |
| -0 | checkstyle | 0m 38s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 264 unchanged - 4 fixed = 266 total (was 268) |
| +1 | mvnsite | 1m 2s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 59s | the patch passed |
| +1 | javadoc | 0m 41s | the patch passed |
| -1 | unit | 67m 54s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 96m 45s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12872144/HADOOP-14502.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux c3eba31d616c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a062374 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 for v002 patch.
Thanks Erik! +1 on v2 patch as well. Tested with MiniHadoopClusterManager and it shows desired behavior.
}, {
"name" : "Hadoop:service=NameNode,name=NameNodeActivity",
"modelerType" : "NameNodeActivity",
"tag.ProcessName" : "NameNode",
"tag.SessionId" : null,
"tag.Context" : "dfs",
"tag.Hostname" : "zezhang-mn1",
"CreateFileOps" : 2,
"FilesCreated" : 12,
"FilesAppended" : 0,
"GetBlockLocations" : 0,
"FilesRenamed" : 0,
"FilesTruncated" : 0,
"GetListingOps" : 1,
"DeleteFileOps" : 0,
"FilesDeleted" : 0,
"FileInfoOps" : 6,
"AddBlockOps" : 2,
"GetAdditionalDatanodeOps" : 0,
"CreateSymlinkOps" : 0,
"GetLinkTargetOps" : 0,
"FilesInGetListingOps" : 0,
"AllowSnapshotOps" : 0,
"DisallowSnapshotOps" : 0,
"CreateSnapshotOps" : 0,
"DeleteSnapshotOps" : 0,
"RenameSnapshotOps" : 0,
"ListSnapshottableDirOps" : 0,
"SnapshotDiffReportOps" : 0,
"BlockReceivedAndDeletedOps" : 2,
"BlockOpsQueued" : 1,
"BlockOpsBatched" : 0,
"TransactionsNumOps" : 24,
"TransactionsAvgTime" : 1.7083333333333333,
"SyncsNumOps" : 14,
"SyncsAvgTime" : 0.2857142857142857,
"TransactionsBatchedInSync" : 10,
"StorageBlockReportNumOps" : 2,
"StorageBlockReportAvgTime" : 3.5,
"CacheReportNumOps" : 0,
"CacheReportAvgTime" : 0.0,
"GenerateEDEKTimeNumOps" : 0,
"GenerateEDEKTimeAvgTime" : 0.0,
"WarmUpEDEKTimeNumOps" : 0,
"WarmUpEDEKTimeAvgTime" : 0.0,
"ResourceCheckTimeNumOps" : 8,
"ResourceCheckTimeAvgTime" : 0.0,
"SafeModeTime" : 1,
"FsImageLoadTime" : 76,
"GetEditNumOps" : 0,
"GetEditAvgTime" : 0.0,
"GetImageNumOps" : 0,
"GetImageAvgTime" : 0.0,
"PutImageNumOps" : 0,
"PutImageAvgTime" : 0.0,
"TotalFileOps" : 11
},
I'm committing to trunk soon. Let's write a short release note?
Please remember to include the JIRA # in the future, thanks!
c22cf004425daa9c350df5e365b0db85b1628b40 Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps. Contributed by Erik Krogen.
Ping Konstantin Shvachko, Zhe Zhang, thoughts?