Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14581

Appending to EC files crashes NameNode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: erasure-coding
    • Labels:
      None

      Description

      org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.prepareFileForAppend(..)@189

      file.recordModification(iip.getLatestSnapshotId());
      file.toUnderConstruction(leaseHolder, clientMachine);
      
      fsn.getLeaseManager().addLease(
      file.getFileUnderConstructionFeature().getClientName(), file.getId());
      
      LocatedBlock ret = null;
      if (!newBlock) {
      if (file.isStriped()) {
      throw new UnsupportedOperationException(
      "Append on EC file without new block is not supported.");
      }

      In this code "UnsupportedOperationException" exception thows after marking file underConstruction. In this case file is opened without any "Open" editlogs, after some time lease manager close this file and add close edit log.

      When SBN tail this edit log, it will fail with this exception.

      2019-06-13 19:17:51,513 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation CloseOp [length=0, inodeId=0, path=/ECtest/********, replication=1, mtime=1560261947480, atime=1560258249117, blockSize=134217728, blocks=[blk_-9223372036854775792_1005], permissions=root:hadoop:rw-r--r--, aclEntries=null, clientName=, clientMachine=, overwrite=false, storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_CLOSE, txid=1363]
      java.io.IOException: File is not under construction: /ECtest/container-executor
      at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:504)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:286)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:181)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:924)
      at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:329)
      at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
      at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:410)
      at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
      at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:485)
      at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
      

        Attachments

        1. HDFS-14581.001.patch
          4 kB
          Surendra Singh Lilhore
        2. HDFS-14581.002.patch
          4 kB
          Surendra Singh Lilhore

          Issue Links

            Activity

              People

              • Assignee:
                surendrasingh Surendra Singh Lilhore
                Reporter:
                surendrasingh Surendra Singh Lilhore
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: