Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10265

OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.1, 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • tools

    Description

      I use OEV tool to convert editlog to xml file, then convert the xml file back to binary editslog file(so that low version NameNode can load edits that generated by higher version NameNode). But when OP_UPDATE_BLOCKS has no BLOCK tag, the OEV tool doesn't handle the case and exits with InvalidXmlException.

      Here is the stack:

      fromXml error decoding opcode null
      {<PATH>{"/tmp/100M3/slive/data/subDir_13/subDir_7/subDir_15/subDir_11/subFile_5"}, <RPC_CALLID>{"-2"}, <RPC_CLIENTID>{},
      <TXID>{"3875711"}}
      Encountered exception. Exiting: no entry found for BLOCK
      org.apache.hadoop.hdfs.util.XMLUtils$InvalidXmlException: no entry found for BLOCK
              at org.apache.hadoop.hdfs.util.XMLUtils$Stanza.getChildren(XMLUtils.java:242)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$UpdateBlocksOp.fromXml(FSEditLogOp.java:908)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.decodeXml(FSEditLogOp.java:3942)
              ...
      

      Here is part of the xml file:

      <RECORD>
        <OPCODE>OP_UPDATE_BLOCKS</OPCODE>
        <DATA>
          <TXID>3875711</TXID>
          <PATH>/tmp/100M3/slive/data/subDir_13/subDir_7/subDir_15/subDir_11/subFile_5</PATH>
          <RPC_CLIENTID></RPC_CLIENTID>
          <RPC_CALLID>-2</RPC_CALLID>
        </DATA>
      </RECORD>
      

      I tracked the NN's log and found those operation:

      0. The file /tmp/100M3/slive/data/subDir_13/subDir_7/subDir_15/subDir_11/subFile_5 is very small and contains only one block.
      1. Client ask NN to add block to the file.
      2. Client failed to write to DN and asked NameNode to abandon block.
      3. NN remove the block and write an OP_UPDATE_BLOCKS to editlog

      Finally NN generated a OP_UPDATE_BLOCKS with no BLOCK tags.

      In FSEditLogOp$UpdateBlocksOp.fromXml, we need to handle the case above.

      Attachments

        1. HDFS-10265-001.patch
          1 kB
          Wan Chang
        2. HDFS-10265-002.patch
          2 kB
          Wan Chang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wanchang Wan Chang
            wanchang Wan Chang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment