Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3215

Block size is logging as zero Even blockrecevied command received by DN

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:

      Description

      Scenario 1
      ==========

      Start NN and DN.
      write file.
      Block size is logging as zero Even blockrecevied command received by DN

      NN log

      2012-03-14 20:23:40,541 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hadoop-create-user.sh.COPYING. BP-1166515020-10.18.40.24-1331736264353 blk_1264419582929433995_1002

      {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXXXXXX:50010|RBW]]}

      2012-03-14 20:24:26,357 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* addStoredBlock: blockMap updated: XXXXXXX:50010 is added to blk_1264419582929433995_1002

      {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXXXXXX:50010|RBW]]}

      size 0

      DN log

      2012-03-14 20:24:17,519 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block BP-1166515020-XXXXXXX-1331736264353:blk_1264419582929433995_1002 src: /XXXXXXX:53141 dest: /XXXXXXX:50010

      2012-03-14 20:24:26,517 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /XXXXXXX:53141, dest: /XXXXXXX:50010, bytes: 512, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1612873957_1, offset: 0, srvID: DS-1639667928-XXXXXXX-50010-1331736284942, blockid: BP-1166515020-XXXXXXX-1331736264353:blk_1264419582929433995_1002, duration: 1286482503

      2012-03-14 20:24:26,517 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1166515020-XXXXXXX-1331736264353:blk_1264419582929433995_1002, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
      2012-03-14 20:24:31,533 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-1166515020-XXXXXXX-1331736264353:blk_1264419582929433995_1002

      1. HDFS-3215.patch
        0.8 kB
        Shinichi Yamashita
      2. HDFS-3215.patch
        4 kB
        Shinichi Yamashita

        Activity

        Hide
        Shinichi Yamashita added a comment -

        BlockInfo is defined as 0 bytes in UNDER_CONSTRUCTION state. This block size is not changed, and it is output as "size 0".
        On the other hand, DataNode sends block size to NameNode in DataNodeProtocol#blockReceivedAndDeleted.
        I think that it comes to be able to output block size in UNDER_CONSTRUCTIONby applying this information to BlockInfo.

        Show
        Shinichi Yamashita added a comment - BlockInfo is defined as 0 bytes in UNDER_CONSTRUCTION state. This block size is not changed, and it is output as "size 0". On the other hand, DataNode sends block size to NameNode in DataNodeProtocol#blockReceivedAndDeleted. I think that it comes to be able to output block size in UNDER_CONSTRUCTIONby applying this information to BlockInfo.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614576/HDFS-3215.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated -2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
        org.apache.hadoop.hdfs.TestFileAppend
        org.apache.hadoop.hdfs.TestEncryptedTransfer
        org.apache.hadoop.hdfs.TestParallelUnixDomainRead
        org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
        org.apache.hadoop.hdfs.TestDatanodeDeath
        org.apache.hadoop.hdfs.TestShortCircuitLocalRead
        org.apache.hadoop.hdfs.TestPipelines
        org.apache.hadoop.hdfs.TestFileAppend2
        org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
        org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend
        org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
        org.apache.hadoop.hdfs.TestDFSShell
        org.apache.hadoop.hdfs.TestCrcCorruption
        org.apache.hadoop.hdfs.TestPersistBlocks
        org.apache.hadoop.hdfs.TestDataTransferProtocol
        org.apache.hadoop.hdfs.TestAppendDifferentChecksum
        org.apache.hadoop.fs.TestSymlinkHdfsFileContext
        org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
        org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
        org.apache.hadoop.hdfs.TestDecommission
        org.apache.hadoop.fs.TestHDFSFileContextMainOperations
        org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
        org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
        org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
        org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
        org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
        org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
        org.apache.hadoop.hdfs.TestFileAppend3
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.TestFileAppend4
        org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp
        org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
        org.apache.hadoop.hdfs.TestHDFSFileSystemContract
        org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
        org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
        org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
        org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestParallelShortCircuitRead
        org.apache.hadoop.hdfs.TestReadWhileWriting
        org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
        org.apache.hadoop.hdfs.TestFileAppendRestart
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.fs.permission.TestStickyBit
        org.apache.hadoop.hdfs.TestSmallBlock
        org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
        org.apache.hadoop.hdfs.TestReplication
        org.apache.hadoop.hdfs.TestDFSUpgrade
        org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5483//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5483//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614576/HDFS-3215.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated -2 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.TestParallelUnixDomainRead org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestPersistBlocks org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.fs.TestSymlinkHdfsFileContext org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead org.apache.hadoop.fs.TestSymlinkHdfsFileSystem org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestParallelShortCircuitRead org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes org.apache.hadoop.hdfs.TestFileAppendRestart org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.TestFileConcurrentReader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5483//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5483//console This message is automatically generated.
        Hide
        Shinichi Yamashita added a comment -

        I changed it to treat block size with an argument based on a previous test result.

        Show
        Shinichi Yamashita added a comment - I changed it to treat block size with an argument based on a previous test result.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614783/HDFS-3215.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5498//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5498//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614783/HDFS-3215.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5498//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5498//console This message is automatically generated.

          People

          • Assignee:
            Shinichi Yamashita
            Reporter:
            Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development