Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7182

JMX metrics aren't accessible when NN is busy

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out "RollingUpgradeStatus" requires FSNamesystem read lock.

      1. HDFS-7182.patch
        6 kB
        Ming Ma
      2. HDFS-7182-2.patch
        6 kB
        Ming Ma
      3. HDFS-7182-3.patch
        6 kB
        Ming Ma

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Sangjin Lee backported this to 2.6.1, after fixing a merge-conflict in TestFSNamesystemMBean. I just pushed the commit after running compilation and TestFSNamesystemMBean which changed in the patch.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1, after fixing a merge-conflict in TestFSNamesystemMBean. I just pushed the commit after running compilation and TestFSNamesystemMBean which changed in the patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2020 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2020/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2020 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2020/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #70 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/70/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #70 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/70/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #66 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/66/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #66 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/66/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2001/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2001/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #803 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/803/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #803 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/803/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #69 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/69/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #69 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/69/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        mingma Ming Ma added a comment -

        Thanks, Jing and Akira!

        Show
        mingma Ming Ma added a comment - Thanks, Jing and Akira!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6838 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6838/)
        HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6838 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6838/ ) HDFS-7182 . JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the contribution, Ming Ma! I've committed this to trunk and branch-2.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the contribution, Ming Ma ! I've committed this to trunk and branch-2.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12691404/HDFS-7182-3.patch
        against trunk revision ae91b13.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9171//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9171//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691404/HDFS-7182-3.patch against trunk revision ae91b13. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9171//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9171//console This message is automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks Ming! +1 pending Jenkins.

        Show
        jingzhao Jing Zhao added a comment - Thanks Ming! +1 pending Jenkins.
        Hide
        mingma Ming Ma added a comment -

        Good catch. Thanks, Jing. Here is the updated patch.

        Show
        mingma Ming Ma added a comment - Good catch. Thanks, Jing. Here is the updated patch.
        Hide
        jingzhao Jing Zhao added a comment -

        Currently we check if rollingUpgradeInfo is null to determine if rolling upgrade is in process. Without the read lock do we need to declare the object as volatile to make sure the jmx returns the correct result?

        Show
        jingzhao Jing Zhao added a comment - Currently we check if rollingUpgradeInfo is null to determine if rolling upgrade is in process. Without the read lock do we need to declare the object as volatile to make sure the jmx returns the correct result?
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1

        Show
        ajisakaa Akira Ajisaka added a comment - +1
        Show
        mingma Ming Ma added a comment - The test errors are known issues, https://issues.apache.org/jira/browse/HADOOP-11432 , https://issues.apache.org/jira/browse/HDFS-7524 and https://issues.apache.org/jira/browse/HDFS-7552 .
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12688194/HDFS-7182-2.patch
        against trunk revision efe6357.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.fs.TestSymlinkHdfsFileContext
        org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
        org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12688194/HDFS-7182-2.patch against trunk revision efe6357. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestSymlinkHdfsFileContext org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9085//console This message is automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Rebased for trunk. Appreciate any input.

        Show
        mingma Ming Ma added a comment - Rebased for trunk. Appreciate any input.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12672609/HDFS-7182.patch
        against trunk revision 0402bad.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9083//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672609/HDFS-7182.patch against trunk revision 0402bad. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9083//console This message is automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Anyone else has suggestions on this? The patch has been running fine in one of our production clusters.

        Show
        mingma Ming Ma added a comment - Anyone else has suggestions on this? The patch has been running fine in one of our production clusters.
        Hide
        mingma Ming Ma added a comment -

        Thanks, Akira.

        Findbugs and failed unit tests aren't related.

        Show
        mingma Ming Ma added a comment - Thanks, Akira. Findbugs and failed unit tests aren't related.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12672609/HDFS-7182.patch
        against trunk revision 29f5200.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
        org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
        org.apache.hadoop.hdfs.TestCrcCorruption
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672609/HDFS-7182.patch against trunk revision 29f5200. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestEditLogRace org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8302//console This message is automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        The patch looks good to me. +1 (non-binding) pending Jenkins.

        Show
        ajisakaa Akira Ajisaka added a comment - The patch looks good to me. +1 (non-binding) pending Jenkins.
        Hide
        mingma Ming Ma added a comment -

        getRollingUpgradeStatus doesn't have to take FSNamesystem's read lock.

        Also fix the unit test so that any new metrics will be covered automatically.

        Show
        mingma Ming Ma added a comment - getRollingUpgradeStatus doesn't have to take FSNamesystem's read lock. Also fix the unit test so that any new metrics will be covered automatically.

          People

          • Assignee:
            mingma Ming Ma
            Reporter:
            mingma Ming Ma
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development