Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14502

Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: metrics
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Remove the BlockReport(NumOps,AvgTime) metrics emitted under the NameNodeActivity context in favor of StorageBlockReport(NumOps,AvgTime) which more accurately represent the metric. Same for the corresponding quantile metrics.

      Description

      Currently the BlockReport(NumOps|AvgTime) metrics emitted under the RpcDetailedActivity context and those emitted under the NameNodeActivity context are actually reporting different things despite having the same name. NameNodeActivity reports the count/time of per storage block reports, whereas RpcDetailedActivity reports the count/time of per datanode block reports. This makes for a confusing experience with two metrics having the same name reporting different values.

      We already have the StorageBlockReportsOps metric under NameNodeActivity. Can we make StorageBlockReport a MutableRate metric and remove NameNodeActivity#BlockReport metric? Open to other suggestions about how to address this as well. The 3.0 release seems a good time to make this incompatible change.

      1. HADOOP-14502.000.patch
        6 kB
        Erik Krogen
      2. HADOOP-14502.001.patch
        7 kB
        Erik Krogen
      3. HADOOP-14502.002.patch
        7 kB
        Erik Krogen

        Issue Links

          Activity

          Hide
          xkrogen Erik Krogen added a comment -

          Ping Konstantin Shvachko, Zhe Zhang, thoughts?

          Show
          xkrogen Erik Krogen added a comment - Ping Konstantin Shvachko , Zhe Zhang , thoughts?
          Hide
          zhz Zhe Zhang added a comment - - edited

          Can we make StorageBlockReport a MutableRate metric and remove NameNodeActivity#BlockReport metric

          This sounds good to me (as a 3.0 change).

          Pinging Andrew Wang for opinion on breaking compatibility in this case.

          Show
          zhz Zhe Zhang added a comment - - edited Can we make StorageBlockReport a MutableRate metric and remove NameNodeActivity#BlockReport metric This sounds good to me (as a 3.0 change). Pinging Andrew Wang for opinion on breaking compatibility in this case.
          Hide
          andrew.wang Andrew Wang added a comment -

          Sounds good to me.

          Show
          andrew.wang Andrew Wang added a comment - Sounds good to me.
          Hide
          xkrogen Erik Krogen added a comment -

          Thanks both for looking, will post a patch soon.

          Show
          xkrogen Erik Krogen added a comment - Thanks both for looking, will post a patch soon.
          Hide
          xkrogen Erik Krogen added a comment - - edited

          Attaching initial patch. I updated the quantile metrics for BlockReports as well since they also reflect individual storages.

          Old metrics output:

           }, {
              "name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010",
              "BlockReportNumOps" : 2,
              "BlockReportAvgTime" : 42.0,
          ...
            }, {
              "name" : "Hadoop:service=NameNode,name=NameNodeActivity",
              "BlockReport60sNumOps" : 0,
              "BlockReport60s50thPercentileLatency" : 0,
              "BlockReport60s75thPercentileLatency" : 0,
              "BlockReport60s90thPercentileLatency" : 0,
              "BlockReport60s95thPercentileLatency" : 0,
              "BlockReport60s99thPercentileLatency" : 0,
              "StorageBlockReportsOps" : 4,
              "BlockReportOps" : 4,
              "BlockReportAvgTime" : 5.5,
          ...
            }, {
          

          New metrics output:

           }, {
              "name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010",
              "BlockReportNumOps" : 2,
              "BlockReportAvgTime" : 42.0,
          ...
            }, {
              "name" : "Hadoop:service=NameNode,name=NameNodeActivity",
              "StorageBlockReport60sNumOps" : 0,
              "StorageBlockReport60s50thPercentileLatency" : 0,
              "StorageBlockReport60s75thPercentileLatency" : 0,
              "StorageBlockReport60s90thPercentileLatency" : 0,
              "StorageBlockReport60s95thPercentileLatency" : 0,
              "StorageBlockReport60s99thPercentileLatency" : 0,
              "StorageBlockReportNumOps" : 4,
              "StorageBlockReportAvgTime" : 5.5,
          ...
            }, {
          

          I also would like to point out this is consistent with the DataNode BlockReport metric:

          }, {
              "name" : "Hadoop:service=DataNode,name=DataNodeActivity-127.0.0.1-58011",
              "BlockReportsNumOps" : 2,
              "BlockReportsAvgTime" : 65.0,
          ...
            }, {
          
          Show
          xkrogen Erik Krogen added a comment - - edited Attaching initial patch. I updated the quantile metrics for BlockReports as well since they also reflect individual storages. Old metrics output: }, { "name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010" , "BlockReportNumOps" : 2, "BlockReportAvgTime" : 42.0, ... }, { "name" : "Hadoop:service=NameNode,name=NameNodeActivity" , "BlockReport60sNumOps" : 0, "BlockReport60s50thPercentileLatency" : 0, "BlockReport60s75thPercentileLatency" : 0, "BlockReport60s90thPercentileLatency" : 0, "BlockReport60s95thPercentileLatency" : 0, "BlockReport60s99thPercentileLatency" : 0, "StorageBlockReportsOps" : 4, "BlockReportOps" : 4, "BlockReportAvgTime" : 5.5, ... }, { New metrics output: }, { "name" : "Hadoop:service=NameNode,name=RpcDetailedActivityForPort58010" , "BlockReportNumOps" : 2, "BlockReportAvgTime" : 42.0, ... }, { "name" : "Hadoop:service=NameNode,name=NameNodeActivity" , "StorageBlockReport60sNumOps" : 0, "StorageBlockReport60s50thPercentileLatency" : 0, "StorageBlockReport60s75thPercentileLatency" : 0, "StorageBlockReport60s90thPercentileLatency" : 0, "StorageBlockReport60s95thPercentileLatency" : 0, "StorageBlockReport60s99thPercentileLatency" : 0, "StorageBlockReportNumOps" : 4, "StorageBlockReportAvgTime" : 5.5, ... }, { I also would like to point out this is consistent with the DataNode BlockReport metric: }, { "name" : "Hadoop:service=DataNode,name=DataNodeActivity-127.0.0.1-58011" , "BlockReportsNumOps" : 2, "BlockReportsAvgTime" : 65.0, ... }, {
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 20s trunk passed
          +1 compile 0m 58s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 242 unchanged - 4 fixed = 246 total (was 246)
          +1 mvnsite 0m 54s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 68m 3s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          94m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872094/HADOOP-14502.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a54307184be9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5672ae7
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 20s trunk passed +1 compile 0m 58s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 1s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 242 unchanged - 4 fixed = 246 total (was 246) +1 mvnsite 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 68m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 94m 48s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872094/HADOOP-14502.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a54307184be9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5672ae7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12482/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          v001 patch to fix unit test failure

          Show
          xkrogen Erik Krogen added a comment - v001 patch to fix unit test failure
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 30s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 264 unchanged - 4 fixed = 268 total (was 268)
          +1 mvnsite 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 87m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          113m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872111/HADOOP-14502.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78a9603618da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a062374
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 30s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 55s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 264 unchanged - 4 fixed = 268 total (was 268) +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 87m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 113m 27s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872111/HADOOP-14502.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78a9603618da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a062374 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12484/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          v002 fixes line length checkstyle... My mistake for not addressing in v2. The two checkstyle about private variables is standard for all of the metrics variables.

          Show
          xkrogen Erik Krogen added a comment - v002 fixes line length checkstyle... My mistake for not addressing in v2. The two checkstyle about private variables is standard for all of the metrics variables.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 44s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 57s the patch passed
          +1 javac 0m 57s the patch passed
          -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 264 unchanged - 4 fixed = 266 total (was 268)
          +1 mvnsite 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 0m 41s the patch passed
          -1 unit 67m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          96m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872144/HADOOP-14502.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c3eba31d616c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a062374
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 44s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 5s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 264 unchanged - 4 fixed = 266 total (was 268) +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 67m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 96m 45s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872144/HADOOP-14502.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c3eba31d616c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a062374 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12486/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vagarychen Chen Liang added a comment -

          +1 for v002 patch.

          Show
          vagarychen Chen Liang added a comment - +1 for v002 patch.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! +1 on v2 patch as well. Tested with MiniHadoopClusterManager and it shows desired behavior.

            }, {
              "name" : "Hadoop:service=NameNode,name=NameNodeActivity",
              "modelerType" : "NameNodeActivity",
              "tag.ProcessName" : "NameNode",
              "tag.SessionId" : null,
              "tag.Context" : "dfs",
              "tag.Hostname" : "zezhang-mn1",
              "CreateFileOps" : 2,
              "FilesCreated" : 12,
              "FilesAppended" : 0,
              "GetBlockLocations" : 0,
              "FilesRenamed" : 0,
              "FilesTruncated" : 0,
              "GetListingOps" : 1,
              "DeleteFileOps" : 0,
              "FilesDeleted" : 0,
              "FileInfoOps" : 6,
              "AddBlockOps" : 2,
              "GetAdditionalDatanodeOps" : 0,
              "CreateSymlinkOps" : 0,
              "GetLinkTargetOps" : 0,
              "FilesInGetListingOps" : 0,
              "AllowSnapshotOps" : 0,
              "DisallowSnapshotOps" : 0,
              "CreateSnapshotOps" : 0,
              "DeleteSnapshotOps" : 0,
              "RenameSnapshotOps" : 0,
              "ListSnapshottableDirOps" : 0,
              "SnapshotDiffReportOps" : 0,
              "BlockReceivedAndDeletedOps" : 2,
              "BlockOpsQueued" : 1,
              "BlockOpsBatched" : 0,
              "TransactionsNumOps" : 24,
              "TransactionsAvgTime" : 1.7083333333333333,
              "SyncsNumOps" : 14,
              "SyncsAvgTime" : 0.2857142857142857,
              "TransactionsBatchedInSync" : 10,
              "StorageBlockReportNumOps" : 2,
              "StorageBlockReportAvgTime" : 3.5,
              "CacheReportNumOps" : 0,
              "CacheReportAvgTime" : 0.0,
              "GenerateEDEKTimeNumOps" : 0,
              "GenerateEDEKTimeAvgTime" : 0.0,
              "WarmUpEDEKTimeNumOps" : 0,
              "WarmUpEDEKTimeAvgTime" : 0.0,
              "ResourceCheckTimeNumOps" : 8,
              "ResourceCheckTimeAvgTime" : 0.0,
              "SafeModeTime" : 1,
              "FsImageLoadTime" : 76,
              "GetEditNumOps" : 0,
              "GetEditAvgTime" : 0.0,
              "GetImageNumOps" : 0,
              "GetImageAvgTime" : 0.0,
              "PutImageNumOps" : 0,
              "PutImageAvgTime" : 0.0,
              "TotalFileOps" : 11
            },
          

          I'm committing to trunk soon. Let's write a short release note?

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! +1 on v2 patch as well. Tested with MiniHadoopClusterManager and it shows desired behavior. }, { "name" : "Hadoop:service=NameNode,name=NameNodeActivity" , "modelerType" : "NameNodeActivity" , "tag.ProcessName" : "NameNode" , "tag.SessionId" : null , "tag.Context" : "dfs" , "tag.Hostname" : "zezhang-mn1" , "CreateFileOps" : 2, "FilesCreated" : 12, "FilesAppended" : 0, "GetBlockLocations" : 0, "FilesRenamed" : 0, "FilesTruncated" : 0, "GetListingOps" : 1, "DeleteFileOps" : 0, "FilesDeleted" : 0, "FileInfoOps" : 6, "AddBlockOps" : 2, "GetAdditionalDatanodeOps" : 0, "CreateSymlinkOps" : 0, "GetLinkTargetOps" : 0, "FilesInGetListingOps" : 0, "AllowSnapshotOps" : 0, "DisallowSnapshotOps" : 0, "CreateSnapshotOps" : 0, "DeleteSnapshotOps" : 0, "RenameSnapshotOps" : 0, "ListSnapshottableDirOps" : 0, "SnapshotDiffReportOps" : 0, "BlockReceivedAndDeletedOps" : 2, "BlockOpsQueued" : 1, "BlockOpsBatched" : 0, "TransactionsNumOps" : 24, "TransactionsAvgTime" : 1.7083333333333333, "SyncsNumOps" : 14, "SyncsAvgTime" : 0.2857142857142857, "TransactionsBatchedInSync" : 10, "StorageBlockReportNumOps" : 2, "StorageBlockReportAvgTime" : 3.5, "CacheReportNumOps" : 0, "CacheReportAvgTime" : 0.0, "GenerateEDEKTimeNumOps" : 0, "GenerateEDEKTimeAvgTime" : 0.0, "WarmUpEDEKTimeNumOps" : 0, "WarmUpEDEKTimeAvgTime" : 0.0, "ResourceCheckTimeNumOps" : 8, "ResourceCheckTimeAvgTime" : 0.0, "SafeModeTime" : 1, "FsImageLoadTime" : 76, "GetEditNumOps" : 0, "GetEditAvgTime" : 0.0, "GetImageNumOps" : 0, "GetImageAvgTime" : 0.0, "PutImageNumOps" : 0, "PutImageAvgTime" : 0.0, "TotalFileOps" : 11 }, I'm committing to trunk soon. Let's write a short release note?
          Hide
          andrew.wang Andrew Wang added a comment -

          Please remember to include the JIRA # in the future, thanks!

          c22cf004425daa9c350df5e365b0db85b1628b40 Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps. Contributed by Erik Krogen.
          
          Show
          andrew.wang Andrew Wang added a comment - Please remember to include the JIRA # in the future, thanks! c22cf004425daa9c350df5e365b0db85b1628b40 Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps. Contributed by Erik Krogen.
          Hide
          zhz Zhe Zhang added a comment -

          Sorry, my bad.

          Show
          zhz Zhe Zhang added a comment - Sorry, my bad.

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development