Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4080

Add a separate logger for block state change logs to enable turning off those logs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.3-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Add a separate logger "BlockStateChange" for block state change logs.

      Description

      Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events.

      It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages.

      1. hdfs-4080-trunk.patch
        24 kB
        Kihwal Lee
      2. hdfs-4080-branch-0.23.patch
        23 kB
        Kihwal Lee
      3. hdfs-4080.patch
        23 kB
        Kihwal Lee
      4. hdfs-4080.1.patch
        32 kB
        Kihwal Lee
      5. hdfs-4080.patch
        33 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1253/)
          Moving HDFS-4080 to Incompatible Change section. (Revision 1407567)
          HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1253/ ) Moving HDFS-4080 to Incompatible Change section. (Revision 1407567) HDFS-4080 . Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1222 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1222/)
          Moving HDFS-4080 to Incompatible Change section. (Revision 1407567)
          HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1222 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1222/ ) Moving HDFS-4080 to Incompatible Change section. (Revision 1407567) HDFS-4080 . Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #431 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/431/)
          HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407582)

          Result = UNSTABLE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407582
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #431 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/431/ ) HDFS-4080 . Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407582) Result = UNSTABLE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407582 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #32 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/32/)
          Moving HDFS-4080 to Incompatible Change section. (Revision 1407567)
          HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #32 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/32/ ) Moving HDFS-4080 to Incompatible Change section. (Revision 1407567) HDFS-4080 . Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk, branch-2 and 0.23.

          Thank you Kihwal.

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk, branch-2 and 0.23. Thank you Kihwal.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2991 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2991/)
          Moving HDFS-4080 to Incompatible Change section. (Revision 1407567)
          HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2991 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2991/ ) Moving HDFS-4080 to Incompatible Change section. (Revision 1407567) HDFS-4080 . Add a separate logger for block state change logs to enable turning off those logs. Contributed by Kihwal Lee. (Revision 1407566) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12552722/hdfs-4080-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3469//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3469//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552722/hdfs-4080-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3469//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3469//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          A new trunk patch and a branch-0.23 patch are attached.

          Show
          Kihwal Lee added a comment - A new trunk patch and a branch-0.23 patch are attached.
          Hide
          Kihwal Lee added a comment -

          On a second thought, making the category shorter may be better, since having to use a different appender to omit or shorten the category is more invasive. I will follow your original suggestion.

          Show
          Kihwal Lee added a comment - On a second thought, making the category shorter may be better, since having to use a different appender to omit or shorten the category is more invasive. I will follow your original suggestion.
          Hide
          Suresh Srinivas added a comment -

          We should add logger config as a part of this jira. That way a single jira tracks the related changes.

          As regards to "org.apache.hadoop.hdfs.StateChange.block" I am okay keeping the name longer. Though the named logger (not based on class) can have any name, the name you have used is similar to a class name. Consider changing the name to ""org.apache.hadoop.hdfs.StateChange.Block" (block->Block).

          Show
          Suresh Srinivas added a comment - We should add logger config as a part of this jira. That way a single jira tracks the related changes. As regards to "org.apache.hadoop.hdfs.StateChange.block" I am okay keeping the name longer. Though the named logger (not based on class) can have any name, the name you have used is similar to a class name. Consider changing the name to ""org.apache.hadoop.hdfs.StateChange.Block" (block->Block).
          Hide
          Kihwal Lee added a comment -

          instead of the logger name org.apache.hadoop.hdfs.StateChange.block, which is a change from the existing name, should be just change it to BlockStateChange. This potentially saves a logging from printing the entire package name?

          I think keeping the hierarchical naming is better. To reduce the amount of log data, the layout can include something like %c

          {2}

          or omit "c" entirely since these lines already have a distinct signature. Do you think it is better to add the logger config with this layout change to hadoop-common-project/hadoop-common/src/main/conf/log4j.properties? If so, as a part of this jira or a separate common one?

          Show
          Kihwal Lee added a comment - instead of the logger name org.apache.hadoop.hdfs.StateChange.block, which is a change from the existing name, should be just change it to BlockStateChange. This potentially saves a logging from printing the entire package name? I think keeping the hierarchical naming is better. To reduce the amount of log data, the layout can include something like %c {2} or omit "c" entirely since these lines already have a distinct signature. Do you think it is better to add the logger config with this layout change to hadoop-common-project/hadoop-common/src/main/conf/log4j.properties? If so, as a part of this jira or a separate common one?
          Hide
          Suresh Srinivas added a comment -

          Kihwal, the patch looks good.

          One question - instead of the logger name org.apache.hadoop.hdfs.StateChange.block, which is a change from the existing name, should be just change it to BlockStateChange. This potentially saves a logging from printing the entire package name?

          This should also be marked incompatible to indicate the change in logger name?

          Show
          Suresh Srinivas added a comment - Kihwal, the patch looks good. One question - instead of the logger name org.apache.hadoop.hdfs.StateChange.block , which is a change from the existing name, should be just change it to BlockStateChange. This potentially saves a logging from printing the entire package name? This should also be marked incompatible to indicate the change in logger name?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551558/hdfs-4080.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3433//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3433//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551558/hdfs-4080.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3433//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3433//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          The new patch addresses the review comment. The scope of the change is limited to the block state changes= messages from BlockManager & friends and the calls through DatanodeProtocol.

          Show
          Kihwal Lee added a comment - The new patch addresses the review comment. The scope of the change is limited to the block state changes= messages from BlockManager & friends and the calls through DatanodeProtocol.
          Hide
          Suresh Srinivas added a comment -

          Kihwal, quick comment, having come out of my own log cleanup work . It is probably a good idea to retain "Block *" and not change to "Dir *". BTW I have removed unnecessary prefixes such "NameNode." from the logs - see HDFS-4122. Sorry you will have to rebase this patch.

          Show
          Suresh Srinivas added a comment - Kihwal, quick comment, having come out of my own log cleanup work . It is probably a good idea to retain "Block *" and not change to "Dir *". BTW I have removed unnecessary prefixes such "NameNode." from the logs - see HDFS-4122 . Sorry you will have to rebase this patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551221/hdfs-4080.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3418//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3418//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551221/hdfs-4080.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3418//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3418//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Updated the patch to the latest trunk.

          Show
          Kihwal Lee added a comment - Updated the patch to the latest trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551200/hdfs-4080.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3417//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551200/hdfs-4080.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3417//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Thanks Nicholas. I think that will address most of problems for now. Once we have locking refinements such as HDFS-3937, we will be able to do what Aaron suggested as well.

          Show
          Kihwal Lee added a comment - Thanks Nicholas. I think that will address most of problems for now. Once we have locking refinements such as HDFS-3937 , we will be able to do what Aaron suggested as well.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Let's put the block-level log in BlockManager state change log (a new log). Then, it can be set independently of the NN state change log.

          Show
          Tsz Wo Nicholas Sze added a comment - Let's put the block-level log in BlockManager state change log (a new log). Then, it can be set independently of the NN state change log.
          Hide
          Kihwal Lee added a comment -

          Many are being logged while the lock is held by an object far up in the call tree, so it is not as simple as HDFS-4052. The ultimate solution may be finer granule locking.

          Show
          Kihwal Lee added a comment - Many are being logged while the lock is held by an object far up in the call tree, so it is not as simple as HDFS-4052 . The ultimate solution may be finer granule locking.
          Hide
          Aaron T. Myers added a comment -

          Would it be possible to just move the logging outside of the FSNS lock? Jing did a similar change recently in HDFS-4052.

          Show
          Aaron T. Myers added a comment - Would it be possible to just move the logging outside of the FSNS lock? Jing did a similar change recently in HDFS-4052 .

            People

            • Assignee:
              Kihwal Lee
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development