Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4160

libhdfs / fuse-dfs should implement O_CREAT | O_EXCL

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • 2.0.3-alpha
    • None
    • libhdfs

    Description

      hdfsOpenFile contains this code:

      if ((flags & O_CREAT) && (flags & O_EXCL)) {                               
        fprintf(stderr, "WARN: hdfs does not truly support O_CREATE && O_EXCL\n");
      } 
      

      But hdfsOpenFile could easily support O_CREAT | O_EXCL by calling FileSystem#create with overwrite = false.

      We should do this. It would also benefit fuse-dfs.

      Attachments

        1. HDFS-4160.001.patch
          9 kB
          Colin McCabe

        Activity

          cmccabe Colin McCabe added a comment -

          This patch implements passing overwrite = false when we're creating a file with O_EXCL.

          I also took the liberty of cleaning up a few other things. The exception matching logic was busted, since it was looking for class names with slashes in them, whereas classNameOfObject returns class names with dots (ultimately this comes from GetMethodID). I wrote this logic originally, and I don't know what I was thinking. Fixing this also lets us get rid of the hack in hdfs_json_parser.c where we convert dots to slashes.

          EIO is also a better error code to return if we don't know the exception we were given than EINTERNAL. The latter should be reserved for things like JNI internal errors, not exceptions.

          I added FileAlreadyExistsException to the list of exceptions we're catching and translating. It translates to EEXIST.

          I added a unit test for O_EXCL in test_libhdfs_threaded. I fixed a few cases where we were printing out test errors without prefixing them with TEST_ERROR (if we don't do this, they get lost in the noise).

          cmccabe Colin McCabe added a comment - This patch implements passing overwrite = false when we're creating a file with O_EXCL . I also took the liberty of cleaning up a few other things. The exception matching logic was busted, since it was looking for class names with slashes in them, whereas classNameOfObject returns class names with dots (ultimately this comes from GetMethodID ). I wrote this logic originally, and I don't know what I was thinking. Fixing this also lets us get rid of the hack in hdfs_json_parser.c where we convert dots to slashes. EIO is also a better error code to return if we don't know the exception we were given than EINTERNAL . The latter should be reserved for things like JNI internal errors, not exceptions. I added FileAlreadyExistsException to the list of exceptions we're catching and translating. It translates to EEXIST. I added a unit test for O_EXCL in test_libhdfs_threaded . I fixed a few cases where we were printing out test errors without prefixing them with TEST_ERROR (if we don't do this, they get lost in the noise).
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4451//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4451//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4451//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4451//console This message is automatically generated.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch
          Optional Tests javac unit
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10541/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch Optional Tests javac unit git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10541/console This message was automatically generated.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch
          Optional Tests javac unit
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10566/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12585242/HDFS-4160.001.patch Optional Tests javac unit git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10566/console This message was automatically generated.

          People

            cmccabe Colin McCabe
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: