Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-777

A zero size file is created when SpaceQuota exceeded

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.20.1
    • None
    • None
    • None

    • Debian GNU/Linux 5.0
      hadoop-0.20.1
      java version "1.6.0_12"
      Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
      Java HotSpot(TM) Server VM (build 11.2-b01, mixed mode)

    • libhdfs

    Description

      The issue can be reproduced by the following steps:

      $ cd hadoop
      $ bin/hadoop fs -mkdir /tmp
      $ bin/hadoop dfsadmin -setSpaceQuota 1m /tmp

      $ bin/hadoop fs -count -q /tmp
      none inf 1048576 1048576 1 0 0 hdfs://debian:9000/tmp

      $ ls -l hadoop-0.20.1-core.jar
      rw-rr- 1 freestyler freestyler 2682112 2009-09-02 04:59 hadoop-0.20.1-core.jar

      $ bin/hadoop fs -put hadoop-0.20.1-core.jar /tmp/test.jar

      09/11/19 12:09:35 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
      at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
      Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:156)
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsInTree(INodeDirectoryWithQuota.java:127)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:859)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:265)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1436)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1285)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

      at org.apache.hadoop.ipc.Client.call(Client.java:739)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
      at $Proxy0.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy0.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
      ... 3 more

      09/11/19 12:09:35 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
      09/11/19 12:09:35 WARN hdfs.DFSClient: Could not get block locations. Source file "/tmp/test.jar" - Aborting...
      put: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m

      Then a zero size file is there which is bad.

      [freestyler@debian hadoop]$ bin/hadoop fs -lsr /tmp
      rw-rr- 2 freestyler supergroup 0 2009-11-19 12:09 /tmp/test.jar

      Even worse is that when I try to write to this '/tmp/test.jar' using libhdfs (see the code below),
      it's stuck on hdfsOpenFile about 2 minutes.

      $ cat test.c

      test.c
      #include "hdfs.h"

int main(int argc, char **argv) {

    hdfsFS fs = hdfsConnect("default", 0);
    if(!fs) {
        fprintf(stderr, "Oops! Failed to connect to hdfs!\n");
        exit(-1);
    }

      const char* writePath = "/tmp/test.jar";
    {
        //Write tests
        fprintf(stderr, "opening %s for writing!\n", writePath);
        hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
        if(!writeFile) {
            fprintf(stderr, "Failed to open %s for writing!\n", writePath);
            exit(-1);
        }
        fprintf(stderr, "Opened %s for writing successfully...\n", writePath);

        char* buffer = "Hello, World!";
        tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
        fprintf(stderr, "Wrote %d bytes\n", num_written_bytes);
        hdfsCloseFile(fs, writeFile);
    }

    hdfsDisconnect(fs);
    return 0;
}

      $ time ./a.out

      opening /tmp/test.jar for writing!
      Opened /tmp/test.jar for writing successfully...
      Wrote 14 bytes
      Nov 19, 2009 12:14:45 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer run
      WARNING: DataStreamer Exception: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
      at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
      Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:156)
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsInTree(INodeDirectoryWithQuota.java:127)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:859)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:265)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1436)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1285)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

      at org.apache.hadoop.ipc.Client.call(Client.java:739)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
      at $Proxy0.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy0.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
      ... 3 more

      Nov 19, 2009 12:14:45 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream processDatanodeError
      WARNING: Error Recovery for block null bad datanode[0] nodes == null
      Nov 19, 2009 12:14:45 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream processDatanodeError
      WARNING: Could not get block locations. Source file "/tmp/test.jar" - Aborting...
      Exception in thread "main" org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
      at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
      Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp is exceeded: quota=1048576 diskspace consumed=128.0m
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:156)
      at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsInTree(INodeDirectoryWithQuota.java:127)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:859)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:265)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1436)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1285)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

      at org.apache.hadoop.ipc.Client.call(Client.java:739)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
      at $Proxy0.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy0.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
      ... 3 more
      Call to org/apache/hadoop/fs/FSDataOutputStream::close failed!

      real 1m0.579s
      user 0m0.404s
      sys 0m0.052s

      And, the above code didn't write successfully.

      $ bin/hadoop fs -lsr /tmp
      rw-rr- 2 freestyler supergroup 0 2009-11-19 12:14 /tmp/test.jar

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-172 Quota exceed exception creates file of size 0

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              freestyler freestyler
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: