Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13164

File not closed if streamer fail with DSQuotaExceededException

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

       This is found during yarn log aggregation but theoretically could happen to any client.

      If the dir's space quota is exceeded, the following would happen when a file is created:

      • client startFile rpc to NN, gets a DFSOutputStream.
      • writing to the stream would trigger the streamer to getAdditionalBlock rpc to NN, which would get the DSQuotaExceededException
      • client closes the stream
         
        The fact that this would leave a 0-sized (or whatever size left in the quota) file in HDFS is beyond the scope of this jira. However, the file would be left in openforwrite status (shown in fsck -openforwrite) at least, and could potentially leak leaseRenewer too.

      This is because in the close implementation,

      1. isClosed is first checked, and the close call will be a no-op if isClosed == true.
      2. flushInternal checks isClosed, and throws the exception right away if true

      isClosed does this: return closed || getStreamer().streamerClosed;

      When the disk quota is reached, getAdditionalBlock will throw when the streamer calls. Because the streamer runs in a separate thread, at the time the client calls close on the stream, the streamer may or may not have reached the Quota exception. If it has, then due to #1, the close call on the stream will be no-op. If it hasn't, then due to #2 the completeFile logic will be skipped.

      protected synchronized void closeImpl() throws IOException {
          if (isClosed()) {
            IOException e = lastException.getAndSet(null);
            if (e == null)
              return;
            else
              throw e;
          }
        try {
          flushBuffer(); // flush from all upper layers
          ...
          flushInternal(); // flush all data to Datanodes
      
          // get last block before destroying the streamer
          ExtendedBlock lastBlock = getStreamer().getBlock();
      
          try (TraceScope ignored =
             dfsClient.getTracer().newScope("completeFile")) {
             completeFile(lastBlock);
          }
         } catch (ClosedChannelException ignored) {
         } finally {
           closeThreads(true);
         }
       }
      
       

      Log snippets:

      2018-02-16 15:59:32,916 DEBUG org.apache.hadoop.hdfs.DFSClient: DataStreamer Quota Exception
      org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /DIR is exceeded: quota = 2000000 B = 1.91 MB but diskspace consumed = 404139552 B = 385.42 MB
              at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:149)
              at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:159)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:2124)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1991)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1966)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:463)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3896)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3484)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:686)
              at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:217)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
              at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
              at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1833)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1626)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:788)
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.DSQuotaExceededException): The DiskSpace quota of /tmp/logs/systest/logs is exceeded: quota = 2000000 B = 1.91 MB but diskspace consumed = 404139552 B = 385.42 MB
              at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:149)
              at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:159)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:2124)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1991)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1966)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:463)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3896)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3484)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:686)
              at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:217)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1504)
              at org.apache.hadoop.ipc.Client.call(Client.java:1441)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy82.addBlock(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:423)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
              at com.sun.proxy.$Proxy83.addBlock(Unknown Source)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1830)
              ... 2 more
      

        Attachments

        1. HDFS-13164.01.patch
          10 kB
          Xiao Chen
        2. HDFS-13164.02.patch
          10 kB
          Xiao Chen
        3. HDFS-13164.branch-2.8.01.patch
          10 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                xiaochen Xiao Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: