Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1117

HDFS portion of HADOOP-6728 (ovehaul metrics framework)

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Metrics names are standardized to use CapitalizedCamelCase. Some examples:
      # Metrics names using "_" is changed to new naming scheme. Eg: bytes_written changes to BytesWritten.
      # All metrics names start with capitals. Example: threadsBlocked changes to ThreadsBlocked.
      Show
      Metrics names are standardized to use CapitalizedCamelCase. Some examples: # Metrics names using "_" is changed to new naming scheme. Eg: bytes_written changes to BytesWritten. # All metrics names start with capitals. Example: threadsBlocked changes to ThreadsBlocked.
    1. HDFS-1117.4.patch
      98 kB
      Suresh Srinivas
    2. HDFS-1117.3.patch
      99 kB
      Suresh Srinivas
    3. HDFS-1117.2.patch
      99 kB
      Suresh Srinivas
    4. HDFS-1117.patch
      96 kB
      Suresh Srinivas

      Issue Links

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #658 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/ )
        Hide
        Suresh Srinivas added a comment -

        I committed the patch. Thank you Luke.

        Show
        Suresh Srinivas added a comment - I committed the patch. Thank you Luke.
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for he patch.

        Show
        Jitendra Nath Pandey added a comment - +1 for he patch.
        Hide
        Suresh Srinivas added a comment -

        Attached patch with minor updates.

        Show
        Suresh Srinivas added a comment - Attached patch with minor updates.
        Hide
        Luke Lu added a comment -

        The patch contains:

        +    System.out.println("SURESH " + cluster.getNamesystem().getCorruptReplicaBlocks());
        +    System.out.println("SURESH " + jmx.getValue("CorruptBlocks"));
        

        which probably should be removed.

        The additional classpath ref is probably redundant for the trunk build.xml.

        Otherwise, +1.

        Show
        Luke Lu added a comment - The patch contains: + System.out.println("SURESH " + cluster.getNamesystem().getCorruptReplicaBlocks()); + System.out.println("SURESH " + jmx.getValue("CorruptBlocks")); which probably should be removed. The additional classpath ref is probably redundant for the trunk build.xml. Otherwise, +1.
        Hide
        Suresh Srinivas added a comment -

        This is porting changes from 203 into trunk. Can some one review the changes I made on top of porting:

        1. Increased junit java heap from 512m to 1024m.
        2. Couple of updated to fix unit test failures.
        Show
        Suresh Srinivas added a comment - This is porting changes from 203 into trunk. Can some one review the changes I made on top of porting: Increased junit java heap from 512m to 1024m. Couple of updated to fix unit test failures.
        Hide
        Suresh Srinivas added a comment -

        TestHDFSCLI, TestDFSStorageStateRecovery are known failures. Other failures are not related to this patch.

        Show
        Suresh Srinivas added a comment - TestHDFSCLI, TestDFSStorageStateRecovery are known failures. Other failures are not related to this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479174/HDFS-1117.3.patch
        against trunk revision 1102833.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 43 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
        org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestLargeBlock

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479174/HDFS-1117.3.patch against trunk revision 1102833. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 43 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestLargeBlock +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/519//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Some tests are failing due to OutOfMemoryError. Increasing the junit java heap size from 512m to 1024m.

        Show
        Suresh Srinivas added a comment - Some tests are failing due to OutOfMemoryError. Increasing the junit java heap size from 512m to 1024m.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12478923/HDFS-1117.2.patch
        against trunk revision 1102513.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 42 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
        org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
        org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
        org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.hdfs.TestLargeBlock

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478923/HDFS-1117.2.patch against trunk revision 1102513. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 42 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestLargeBlock +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/509//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Updated patch fixes unit test failures related to this patch.

        Show
        Suresh Srinivas added a comment - Updated patch fixes unit test failures related to this patch.
        Hide
        Tanping Wang added a comment -

        Just a quick note that TestJMXGet is failing on HDFS trunk since yesterday. This patch should fix the failutre. Expecting test failure of TestJMXGet to be fixed after this patch.

        Show
        Tanping Wang added a comment - Just a quick note that TestJMXGet is failing on HDFS trunk since yesterday. This patch should fix the failutre. Expecting test failure of TestJMXGet to be fixed after this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12478750/HDFS-1117.patch
        against trunk revision 1101675.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 36 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        org.apache.hadoop.hdfs.TestDFSShell
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.hdfs.TestHDFSServerPorts

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478750/HDFS-1117.patch against trunk revision 1101675. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestHDFSServerPorts +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/478//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12478750/HDFS-1117.patch
        against trunk revision 1101675.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 36 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        org.apache.hadoop.hdfs.TestDFSShell
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.hdfs.TestHDFSServerPorts

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478750/HDFS-1117.patch against trunk revision 1101675. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestHDFSServerPorts +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/476//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Porting the metrics2 change from 0.20.203 to trunk.

        Show
        Suresh Srinivas added a comment - Porting the metrics2 change from 0.20.203 to trunk.

          People

          • Assignee:
            Luke Lu
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development