Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6434

Default permission for creating file should be 644 for WebHdfs/HttpFS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: webhdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      The default permissions of files and directories created via WebHDFS and HttpFS are now 644 and 755 respectively. See HDFS-10488 for related discussion.
      Show
      The default permissions of files and directories created via WebHDFS and HttpFS are now 644 and 755 respectively. See HDFS-10488 for related discussion.

      Description

      Creating a file by using WebHdfs without specify permission. file is created with permission 755. it should be 644.
      WebHdfs seems using the same default permission for both file and directory.

      1. HDFS-6434.patch
        5 kB
        Wellington Chevreuil
      2. HDFS-6434.002.patch
        10 kB
        Wellington Chevreuil

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10028 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10028/)
        HDFS-6434. Default permission for creating file should be 644 for (wang: rev c0829f449337b78ac0b995e216f7324843e74dd2)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsCreatePermissions.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/PermissionParam.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10028 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10028/ ) HDFS-6434 . Default permission for creating file should be 644 for (wang: rev c0829f449337b78ac0b995e216f7324843e74dd2) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsCreatePermissions.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/PermissionParam.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks for the contribution Wellington!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for the contribution Wellington!
        Hide
        andrew.wang Andrew Wang added a comment -

        Sorry for the delay, been travelling recently. Thanks for the pointer to HDFS-10488, that reasoning makes sense to me. I'm +1, will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - Sorry for the delay, been travelling recently. Thanks for the pointer to HDFS-10488 , that reasoning makes sense to me. I'm +1, will commit shortly.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Hi, any additional thoughts on my previous comments?

        Show
        wchevreuil Wellington Chevreuil added a comment - Hi, any additional thoughts on my previous comments?
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Hi Andrew Wang, thanks for the comments. Per the umask and FsPermissions methods suggestion, this has been discussed on HDFS-10488, where original proposal was to apply same behaviour to WebHDFS.

        However, conclusions were that it should not apply any umask, but the default permissions for files and directories should be fixed as suggested here

        Show
        wchevreuil Wellington Chevreuil added a comment - Hi Andrew Wang , thanks for the comments. Per the umask and FsPermissions methods suggestion, this has been discussed on HDFS-10488 , where original proposal was to apply same behaviour to WebHDFS. However, conclusions were that it should not apply any umask, but the default permissions for files and directories should be fixed as suggested here
        Hide
        andrew.wang Andrew Wang added a comment -

        Forgot to add, those whitespace issues are unrelated, separate issue that should be fixed now.

        Show
        andrew.wang Andrew Wang added a comment - Forgot to add, those whitespace issues are unrelated, separate issue that should be fixed now.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hmm, so I dug in a little more to this issue, and noticed HADOOP-9155. It looks like the default permissions are supposed to be 0777 for dirs and 0666 for files, and WebHdfsFileSystem is unfortunately not using the new FsPermission methods that were introduced. The default umask of 0022 at CommonConfigurationKeys.FS_PERMISSIONS_UMASK_DEFAULT is what then brings it down to to 0755 or 0644.

        So, is the right fix here to use the new FsPermission methods, i.e. getDirDefault and getFileDefault? I think we can just call this one a bug, given that the behavior in HADOOP-9155 is the expected one.

        Show
        andrew.wang Andrew Wang added a comment - Hmm, so I dug in a little more to this issue, and noticed HADOOP-9155 . It looks like the default permissions are supposed to be 0777 for dirs and 0666 for files, and WebHdfsFileSystem is unfortunately not using the new FsPermission methods that were introduced. The default umask of 0022 at CommonConfigurationKeys.FS_PERMISSIONS_UMASK_DEFAULT is what then brings it down to to 0755 or 0644. So, is the right fix here to use the new FsPermission methods, i.e. getDirDefault and getFileDefault? I think we can just call this one a bug, given that the behavior in HADOOP-9155 is the expected one.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Hi Andrew Wang, thanks for reviewing it. For the symlinks permissions, I had followed what 's described on linux manpages for symlink:

        On Linux, the permissions of a symbolic link are not used in any
               operations; the permissions are always 0777 (read, write, and execute
               for all user categories), and can't be changed.
        

        I'm attaching a new patch with the 2 checkstyle issues fixed. As per the whitespace issue reported, I'm a bit confused, since it seems to be reporting white spaces for a file that is not included in this patch:

        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:210:licenses. 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:219: * Redistribution and use in source and binary forms, with or 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:220: * without modification, are permitted provided that the following 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:222: *  - Redistributions of source code must retain the above copyright 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:224: *  - Redistributions in binary form must reproduce the above copyright 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:225: *    notice, this list of conditions and the following disclaimer in 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:228: *    nor the names of its contributors may be used to endorse or 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:229: *    promote products derived from this software without specific prior 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:231: *    
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:232: * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:233: * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:234: * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:235: * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:236: * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:237: * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:238: * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:239: * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:240: * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:241: * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 
        ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:242: * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED O
        

        Is it possible this is some build process issue?

        Show
        wchevreuil Wellington Chevreuil added a comment - Hi Andrew Wang , thanks for reviewing it. For the symlinks permissions, I had followed what 's described on linux manpages for symlink : On Linux, the permissions of a symbolic link are not used in any operations; the permissions are always 0777 (read, write, and execute for all user categories), and can't be changed. I'm attaching a new patch with the 2 checkstyle issues fixed. As per the whitespace issue reported, I'm a bit confused, since it seems to be reporting white spaces for a file that is not included in this patch: ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:210:licenses. ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:219: * Redistribution and use in source and binary forms, with or ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:220: * without modification, are permitted provided that the following ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:222: * - Redistributions of source code must retain the above copyright ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:224: * - Redistributions in binary form must reproduce the above copyright ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:225: * notice, this list of conditions and the following disclaimer in ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:228: * nor the names of its contributors may be used to endorse or ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:229: * promote products derived from this software without specific prior ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:231: * ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:232: * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:233: * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:234: * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:235: * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:236: * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:237: * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:238: * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:239: * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:240: * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:241: * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt:242: * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED O Is it possible this is some build process issue?
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM overall, thanks for picking this one up Wellington.

        Question, are symlink permissions meaningful? They don't do anything on local FSs. I think it's okay to pass it for consistency, but this change might not be necessary. Up to you.

        Otherwise, we just fixup the checkstyle and can get this in. If you could edit the JIRA to add a release note (since this is an incompatible change) that'd also be appreciated.

        Show
        andrew.wang Andrew Wang added a comment - LGTM overall, thanks for picking this one up Wellington. Question, are symlink permissions meaningful? They don't do anything on local FSs. I think it's okay to pass it for consistency, but this change might not be necessary. Up to you. Otherwise, we just fixup the checkstyle and can get this in. If you could edit the JIRA to add a release note (since this is an incompatible change) that'd also be appreciated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 8m 27s trunk passed
        +1 compile 1m 26s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 33s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 3m 13s trunk passed
        +1 javadoc 1m 22s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 1m 26s the patch passed
        +1 javac 1m 26s the patch passed
        -1 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 2 new + 125 unchanged - 0 fixed = 127 total (was 125)
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 22s the patch passed
        -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 3m 22s the patch passed
        +1 javadoc 1m 11s the patch passed
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
        +1 unit 74m 39s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        104m 40s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809850/HDFS-6434.patch
        JIRA Issue HDFS-6434
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fce86adb151a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 28b66ae
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15750/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15750/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15750/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15750/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 8m 27s trunk passed +1 compile 1m 26s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 1m 22s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 1m 26s the patch passed +1 javac 1m 26s the patch passed -1 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 2 new + 125 unchanged - 0 fixed = 127 total (was 125) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 22s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 22s the patch passed +1 javadoc 1m 11s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 74m 39s hadoop-hdfs in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 104m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809850/HDFS-6434.patch JIRA Issue HDFS-6434 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fce86adb151a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28b66ae Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15750/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15750/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15750/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15750/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Initial patch version with the proposed changes to create files with 644 permissions as default.

        Additional tests created/changed to verify this behaviour.

        Show
        wchevreuil Wellington Chevreuil added a comment - Initial patch version with the proposed changes to create files with 644 permissions as default. Additional tests created/changed to verify this behaviour.

          People

          • Assignee:
            wchevreuil Wellington Chevreuil
            Reporter:
            jyu@cloudera.com Juan Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development