Hadoop Common
  1. Hadoop Common
  2. HADOOP-7223

FileContext createFlag combinations during create are not clearly defined

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During file creation with FileContext, the expected behavior is not clearly defined for combination of createFlag EnumSet.

      1. HADOOP-7223.1.patch
        16 kB
        Suresh Srinivas
      2. HADOOP-7223.2.patch
        16 kB
        Suresh Srinivas
      3. HADOOP-7223.4.patch
        18 kB
        Suresh Srinivas
      4. HADOOP-7223.patch
        14 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          Currently all the comination of createFlag and the expected behavior is not clearly defined in FileContext#create(). Here is my proposal on how the combination must behave the behavior a user needs:

          CreateFlag specifies the file create semantic. Users can combine flags like:
          EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND)

          Use the CreateFlag as follows:
          CREATE - to create a file if it does not exist, else throw FileAlreadyExists.
          APPEND - to append to a file if it exists, else throw FileNotFoundException.
          OVERWRITE - to truncate a file if it exists, else throw FileNotFoundException.
          CREATE|APPEND - to create a file if it does not exist, else append to an existing file.
          CREATE|OVERWRITE - to create a file if it does not exist, else overwrite an existing file.

          Following combination is not valid and will result in HadoopIllegalArgumentException:
          APPEND|OVERWRITE
          CREATE|APPEND|OVERWRITE

          Show
          Suresh Srinivas added a comment - Currently all the comination of createFlag and the expected behavior is not clearly defined in FileContext#create(). Here is my proposal on how the combination must behave the behavior a user needs: CreateFlag specifies the file create semantic. Users can combine flags like: EnumSet.of(CreateFlag.CREATE, CreateFlag.APPEND) Use the CreateFlag as follows: CREATE - to create a file if it does not exist, else throw FileAlreadyExists. APPEND - to append to a file if it exists, else throw FileNotFoundException. OVERWRITE - to truncate a file if it exists, else throw FileNotFoundException. CREATE|APPEND - to create a file if it does not exist, else append to an existing file. CREATE|OVERWRITE - to create a file if it does not exist, else overwrite an existing file. Following combination is not valid and will result in HadoopIllegalArgumentException: APPEND|OVERWRITE CREATE|APPEND|OVERWRITE
          Hide
          Suresh Srinivas added a comment -

          Attached patch makes the following changes:

          1. Added test for various CreateFlag combinations
          2. Removed redundant RawLocalFileSystem#primitiveCreate(), as the based implementation FileSystem#primitiveCreate() is identical.
          3. Added documentation and validation in CreateFlag
          Show
          Suresh Srinivas added a comment - Attached patch makes the following changes: Added test for various CreateFlag combinations Removed redundant RawLocalFileSystem#primitiveCreate(), as the based implementation FileSystem#primitiveCreate() is identical. Added documentation and validation in CreateFlag
          Hide
          Sanjay Radia added a comment -

          +1, however I have the following minor suggestion for improvement.

          Change the tests for exception to take advantage of Junit's @Test(expected=xxxx ...).
          This will force you to create separate methods for each of the tests but it is easier to maintain.

          Show
          Sanjay Radia added a comment - +1, however I have the following minor suggestion for improvement. Change the tests for exception to take advantage of Junit's @Test(expected=xxxx ...). This will force you to create separate methods for each of the tests but it is easier to maintain.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          We should also check for empty set. I know you got it already.

          Show
          Tsz Wo Nicholas Sze added a comment - We should also check for empty set. I know you got it already.
          Hide
          Suresh Srinivas added a comment -

          Both comments taken care of in this new patch.

          Show
          Suresh Srinivas added a comment - Both comments taken care of in this new patch.
          Hide
          Sanjay Radia added a comment -

          Thanks. And thanks for documenting the flags and cleaning up the exceptions and tests.

          Show
          Sanjay Radia added a comment - Thanks. And thanks for documenting the flags and cleaning up the exceptions and tests.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476150/HADOOP-7223.1.patch
          against trunk revision 1090485.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476150/HADOOP-7223.1.patch against trunk revision 1090485. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/344//console This message is automatically generated.
          Hide
          Sanjay Radia added a comment -

          +1

          Show
          Sanjay Radia added a comment - +1
          Hide
          Suresh Srinivas added a comment -

          Updated patch.

          Show
          Suresh Srinivas added a comment - Updated patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476180/HADOOP-7223.2.patch
          against trunk revision 1090485.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476180/HADOOP-7223.2.patch against trunk revision 1090485. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/345//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch.

          Show
          Suresh Srinivas added a comment - I committed the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #548 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/548/)
          HADOOP-7223. FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #548 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/548/ ) HADOOP-7223 . FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #660 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/660/)
          HADOOP-7223. FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #660 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/660/ ) HADOOP-7223 . FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #551 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/551/)
          Undoing commit r1091613 for HADOOP-7223. FileContext createFlag combinations are not clearly defined.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #551 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/551/ ) Undoing commit r1091613 for HADOOP-7223 . FileContext createFlag combinations are not clearly defined.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #661 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/661/)
          Undoing commit r1091613 for HADOOP-7223. FileContext createFlag combinations are not clearly defined.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #661 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/661/ ) Undoing commit r1091613 for HADOOP-7223 . FileContext createFlag combinations are not clearly defined.
          Hide
          Suresh Srinivas added a comment -

          Modified patch:

          1. Changed the file names created in the tests to that of test case name for easier debugging.
          2. CreateFlag OVERWRITE meant CREATE|OVERWRITE in the earlier version. Changed those instances to use OVERWRITE|CREATE.
          Show
          Suresh Srinivas added a comment - Modified patch: Changed the file names created in the tests to that of test case name for easier debugging. CreateFlag OVERWRITE meant CREATE|OVERWRITE in the earlier version. Changed those instances to use OVERWRITE|CREATE.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1

          Show
          Tsz Wo Nicholas Sze added a comment - +1
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476379/HADOOP-7223.4.patch
          against trunk revision 1092519.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 16 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476379/HADOOP-7223.4.patch against trunk revision 1092519. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/348//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #554 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/554/)
          HADOOP-7223. FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #554 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/554/ ) HADOOP-7223 . FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #662 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/662/)
          HADOOP-7223. FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #662 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/662/ ) HADOOP-7223 . FileContext createFlag combinations are not clearly defined. Contributed by Suresh Srinivas.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch.

          Show
          Suresh Srinivas added a comment - I committed the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Renaming Hadoop to Jaadoop? Sounds fun.

          In CHANGES.txt,

          -Hadoop Change Log
          +Hn jaadoop Change Log
          
          Show
          Tsz Wo Nicholas Sze added a comment - Renaming Hadoop to Jaadoop? Sounds fun. In CHANGES.txt, -Hadoop Change Log +Hn jaadoop Change Log
          Hide
          Suresh Srinivas added a comment -

          Good catch Fixed it.

          Show
          Suresh Srinivas added a comment - Good catch Fixed it.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development