Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.6.4
    • 3.0.0-alpha1
    • fs
    • None
    • Incompatible change
    • Hide
      Hdfs dfs chmod command will reset sticky bit permission on a file/directory when the leading sticky bit is omitted in the octal mode (like 644). So when a file/directory permission is applied using octal mode and sticky bit permission needs to be preserved, then it has to be explicitly mentioned in the permission bits (like 1644). This behavior is similar to many other filesystems on Linux/BSD.
      Show
      Hdfs dfs chmod command will reset sticky bit permission on a file/directory when the leading sticky bit is omitted in the octal mode (like 644). So when a file/directory permission is applied using octal mode and sticky bit permission needs to be preserved, then it has to be explicitly mentioned in the permission bits (like 1644). This behavior is similar to many other filesystems on Linux/BSD.

    Description

      When a directory permission is modified using hdfs dfs chmod command and when octal/numeric format is used, the leading sticky bit is not fully honored.

      1. Create a dir dir_test_with_sticky_bit
      2. Apply sticky bit permission on the dir : hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
      3. Remove sticky bit permission on the dir: hdfs dfs -chmod 755 /dir_test_with_sticky_bit

      Expected: Remove the sticky bit on the dir, as it happens on Mac/Linux native filesystem with native chmod.

      4. However, removing sticky bit permission by explicitly turning off the bit works. hdfs dfs -chmod 0755 /dir_test_with_sticky_bit

      manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
      manoj@~/work/hadev-pp: hdfs dfs -ls /
      Found 2 items
      drwxr-xr-t   - manoj supergroup          0 2016-07-25 11:42 /dir_test_with_sticky_bit
      drwxr-xr-x   - manoj supergroup          0 2016-07-25 11:42 /user
      
      manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /dir_test_with_sticky_bit
      manoj@~/work/hadev-pp: hdfs dfs -ls /
      Found 2 items
      drwxr-xr-t   - manoj supergroup          0 2016-07-25 11:42 /dir_test_with_sticky_bit  <=== sticky bit still intact
      drwxr-xr-x   - manoj supergroup          0 2016-07-25 11:42 /user
      
      manoj@~/work/hadev-pp: hdfs dfs -chmod 0755 /dir_test_with_sticky_bit
      manoj@~/work/hadev-pp: hdfs dfs -ls /
      Found 2 items
      drwxr-xr-x   - manoj supergroup          0 2016-07-25 11:42 /dir_test_with_sticky_bit
      drwxr-xr-x   - manoj supergroup          0 2016-07-25 11:42 /user
      manoj@~/work/hadev-pp: 
      

      Attachments

        1. HDFS-10689.004.patch
          9 kB
          Manoj Govindassamy
        2. HDFS-10689.003.patch
          9 kB
          Manoj Govindassamy
        3. HDFS-10689.002.patch
          9 kB
          Manoj Govindassamy
        4. HDFS-10689.001.patch
          11 kB
          Manoj Govindassamy

        Activity

          manojg Manoj Govindassamy made changes -
          Release Note Hdfs dfs chmod command will reset sticky bit permission on a file/directory when the leading sticky bit is omitted in the octal mode (like 644). So when a file/directory permission is applied using octal mode and sticky bit permission needs to be preserved, then it has to be explicitly mentioned in the permission bits (like 1644). This behavior is similar to many other filesystems on Linux/BSD.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10172 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10172/)
          HDFS-10689. Hdfs dfs chmod should reset sticky bit permission when the (lei: rev a3d0cba81071977ffc22a42f60b2ca07da25ae15)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/permission/TestStickyBit.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/PermissionParser.java
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10172 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10172/ ) HDFS-10689 . Hdfs dfs chmod should reset sticky bit permission when the (lei: rev a3d0cba81071977ffc22a42f60b2ca07da25ae15) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/permission/TestStickyBit.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/PermissionParser.java
          eddyxu Lei (Eddy) Xu made changes -
          Fix Version/s 3.0.0-alpha1 [ 12335732 ]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          eddyxu Lei (Eddy) Xu added a comment -

          Committed to trunk.

          eddyxu Lei (Eddy) Xu added a comment - Committed to trunk.

          TestLargeBlockReport, TestDataNodeHotSwapVolumes and TestHttpServerLifecycle test failures are not related to the patch.

          manojg Manoj Govindassamy added a comment - TestLargeBlockReport, TestDataNodeHotSwapVolumes and TestHttpServerLifecycle test failures are not related to the patch.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 7m 22s trunk passed
          +1 checkstyle 1m 30s trunk passed
          +1 mvnsite 1m 50s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 51s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 55s the patch passed
          +1 compile 8m 51s the patch passed
          +1 javac 8m 51s the patch passed
          +1 checkstyle 1m 36s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
          +1 mvnsite 2m 15s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 51s the patch passed
          -1 unit 19m 39s hadoop-common in the patch failed.
          -1 unit 78m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          143m 30s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820753/HDFS-10689.004.patch
          JIRA Issue HDFS-10689
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a7bfbddc65f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 26de4f0
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16241/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16241/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16241/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 7m 22s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 50s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 51s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 8m 51s the patch passed +1 javac 8m 51s the patch passed +1 checkstyle 1m 36s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221) +1 mvnsite 2m 15s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 51s the patch passed -1 unit 19m 39s hadoop-common in the patch failed. -1 unit 78m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 143m 30s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820753/HDFS-10689.004.patch JIRA Issue HDFS-10689 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a7bfbddc65f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 26de4f0 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16241/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16241/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16241/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          eddyxu Lei (Eddy) Xu added a comment -

          +1 pending Jenkins.

          Thanks a lot for working on this, Manoj Govindassamy

          eddyxu Lei (Eddy) Xu added a comment - +1 pending Jenkins. Thanks a lot for working on this, Manoj Govindassamy
          manojg Manoj Govindassamy made changes -
          Attachment HDFS-10689.004.patch [ 12820753 ]

          Attaching v004 patch - avoiding the usage of "*" in imports, and retaining imports old order.

          manojg Manoj Govindassamy added a comment - Attaching v004 patch - avoiding the usage of "*" in imports, and retaining imports old order.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 14m 2s Maven dependency ordering for branch
          +1 mvninstall 8m 26s trunk passed
          +1 compile 8m 31s trunk passed
          +1 checkstyle 1m 48s trunk passed
          +1 mvnsite 2m 17s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 3m 42s trunk passed
          +1 javadoc 2m 3s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 49s the patch passed
          +1 compile 8m 36s the patch passed
          +1 javac 8m 36s the patch passed
          +1 checkstyle 1m 40s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
          +1 mvnsite 2m 5s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 46s the patch passed
          +1 javadoc 1m 57s the patch passed
          -1 unit 22m 10s hadoop-common in the patch failed.
          -1 unit 74m 44s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          177m 46s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820608/HDFS-10689.003.patch
          JIRA Issue HDFS-10689
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 56881d56a349 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b43de80
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16224/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16224/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16224/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16224/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 14m 2s Maven dependency ordering for branch +1 mvninstall 8m 26s trunk passed +1 compile 8m 31s trunk passed +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 17s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 3m 42s trunk passed +1 javadoc 2m 3s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 49s the patch passed +1 compile 8m 36s the patch passed +1 javac 8m 36s the patch passed +1 checkstyle 1m 40s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221) +1 mvnsite 2m 5s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 46s the patch passed +1 javadoc 1m 57s the patch passed -1 unit 22m 10s hadoop-common in the patch failed. -1 unit 74m 44s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 177m 46s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820608/HDFS-10689.003.patch JIRA Issue HDFS-10689 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 56881d56a349 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b43de80 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16224/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16224/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16224/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16224/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          Unit test failures are unrelated to this patch.

          manojg Manoj Govindassamy added a comment - Unit test failures are unrelated to this patch.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 9m 30s Maven dependency ordering for branch
          +1 mvninstall 7m 24s trunk passed
          +1 compile 8m 12s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 2m 2s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 27s trunk passed
          +1 javadoc 1m 51s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 7m 46s the patch passed
          +1 javac 7m 46s the patch passed
          +1 checkstyle 1m 34s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 1m 45s the patch passed
          +1 unit 8m 20s hadoop-common in the patch passed.
          -1 unit 74m 11s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          137m 38s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820608/HDFS-10689.003.patch
          JIRA Issue HDFS-10689
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b424b1192cd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b43de80
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16226/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16226/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16226/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 9m 30s Maven dependency ordering for branch +1 mvninstall 7m 24s trunk passed +1 compile 8m 12s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 2m 2s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 1m 51s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 7m 46s the patch passed +1 javac 7m 46s the patch passed +1 checkstyle 1m 34s root: The patch generated 0 new + 219 unchanged - 2 fixed = 219 total (was 221) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 45s the patch passed +1 unit 8m 20s hadoop-common in the patch passed. -1 unit 74m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 137m 38s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820608/HDFS-10689.003.patch JIRA Issue HDFS-10689 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b424b1192cd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b43de80 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16226/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16226/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16226/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          manojg Manoj Govindassamy made changes -
          Summary "hdfs dfs -chmod 777" does not remove sticky bit Hdfs dfs chmod should reset sticky bit permission when the bit is omitted in the octal mode
          manojg Manoj Govindassamy made changes -
          Attachment HDFS-10689.003.patch [ 12820608 ]

          Fixed checkstyle issues.

          manojg Manoj Govindassamy added a comment - Fixed checkstyle issues.
          manojg Manoj Govindassamy made changes -
          Attachment HDFS-10689.002.patch [ 12820602 ]

          Thanks for the review Lei Eddy.

          1. I looked at few methods that are in the code path of chmod and cleaned them up as well. The changes in FsShellPermission.java. ChmodParser.java, UmaskParsser.java include usage of proper camelCase variable names, removal of unwanted lines, typos etc., But for now reverted these changes to make the patch more focussed.
          2. Yes, PermissionParser.java#73 is formatting only. No change in Matcher Groups. For now, reverted this change.
          3. In PermissionParser#applyOctalPattern: we need to assign '=' to all type as this is apply of new permission and all modes gets a new assignment.
          4. Cleaned up imports in TestStickyBit.java. I am using Intellij's organize imports tool and it reorganized a little more now like arranging them chronologically. Hope its ok.
          5. Removed the tear down section and the cluster restart in TestStickyBit.java
          6. Additionally, made few more changes to suffice checkstyle .

          Attached the v002 patch with review comments incorporated. Please take a look.

          manojg Manoj Govindassamy added a comment - Thanks for the review Lei Eddy. 1. I looked at few methods that are in the code path of chmod and cleaned them up as well. The changes in FsShellPermission.java . ChmodParser.java , UmaskParsser.java include usage of proper camelCase variable names, removal of unwanted lines, typos etc., But for now reverted these changes to make the patch more focussed. 2. Yes, PermissionParser.java#73 is formatting only. No change in Matcher Groups. For now, reverted this change. 3. In PermissionParser#applyOctalPattern: we need to assign '=' to all type as this is apply of new permission and all modes gets a new assignment. 4. Cleaned up imports in TestStickyBit.java. I am using Intellij's organize imports tool and it reorganized a little more now like arranging them chronologically. Hope its ok. 5. Removed the tear down section and the cluster restart in TestStickyBit.java 6. Additionally, made few more changes to suffice checkstyle . Attached the v002 patch with review comments incorporated. Please take a look.

          Thanks for the inputs Chris Nauroth.
          Andrew Wang, sure will change the summary and add a release notes.

          manojg Manoj Govindassamy added a comment - Thanks for the inputs Chris Nauroth . Andrew Wang , sure will change the summary and add a release notes.
          andrew.wang Andrew Wang made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Target Version/s 3.0.0-alpha1 [ 12335732 ]
          Affects Version/s 2.6.4 [ 12334241 ]
          Description
          When a directory permission is modified using hdfs dfs chmod command and when octal/numeric format is used, the leading sticky bit is not fully honored.

          1. Create a dir dir_test_with_sticky_bit
          2. Apply sticky bit permission on the dir : hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
          3. Remove sticky bit permission on the dir: hdfs dfs -chmod 755 /dir_test_with_sticky_bit

          Expected: Remove the sticky bit on the dir, as it happens on Mac/Linux native filesystem with native chmod.

          4. However, removing sticky bit permission by explicitly turning off the bit works. hdfs dfs -chmod 0755 /dir_test_with_sticky_bit

          {noformat}
          manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user

          manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit <=== sticky bit still intact
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user

          manoj@~/work/hadev-pp: hdfs dfs -chmod 0755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user
          manoj@~/work/hadev-pp:
          {noformat}


          When a directory permission is modified using hdfs dfs chmod command and when octal/numeric format is used, the leading sticky bit is not fully honored.

          1. Create a dir dir_test_with_sticky_bit
          2. Apply sticky bit permission on the dir : hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
          3. Remove sticky bit permission on the dir: hdfs dfs -chmod 755 /dir_test_with_sticky_bit

          Expected: Remove the sticky bit on the dir, as it happens on Mac/Linux native filesystem with native chmod.

          4. However, removing sticky bit permission by explicitly turning off the bit works. hdfs dfs -chmod 0755 /dir_test_with_sticky_bit

          {noformat}
          manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user

          manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit <=== sticky bit still intact
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user

          manoj@~/work/hadev-pp: hdfs dfs -chmod 0755 /dir_test_with_sticky_bit
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /dir_test_with_sticky_bit
          drwxr-xr-x - manoj supergroup 0 2016-07-25 11:42 /user
          manoj@~/work/hadev-pp:
          {noformat}


          andrew.wang Andrew Wang added a comment -

          Got it, thanks Chris. Updating the JIRA fields as appropriate.

          Manoj, do you mind adding a little release note about how this change will affect end users? Could also consider changing the summary to say what this patch will do, rather than just what is broken.

          andrew.wang Andrew Wang added a comment - Got it, thanks Chris. Updating the JIRA fields as appropriate. Manoj, do you mind adding a little release note about how this change will affect end users? Could also consider changing the summary to say what this patch will do, rather than just what is broken.
          cnauroth Chris Nauroth added a comment -

          Now the question is if we declare this a bug fix that can be backported to branch-2, or if this behavior change is too incompatible. Given that sticky bits are pretty rare in general, I think it's safe for branch-2, but would welcome other's thoughts. Anything to add Chris Nauroth?

          Andrew Wang, thanks for the notification. I agree with the proposed change, but the compatibility aspects of changes like this are always tricky to consider. In this case, the change is something that potentially weakens authorization. If a user has some automation that runs chmod on a directory, and that user expects the current behavior that sticky bit is preserved, then the effect would be to start allowing users to delete files owned by someone else. Admittedly, sticky bit usage is rare, typically only on /tmp, but I'd still be more comfortable with this as a 3.x change flagged backward-incompatible.

          cnauroth Chris Nauroth added a comment - Now the question is if we declare this a bug fix that can be backported to branch-2, or if this behavior change is too incompatible. Given that sticky bits are pretty rare in general, I think it's safe for branch-2, but would welcome other's thoughts. Anything to add Chris Nauroth? Andrew Wang , thanks for the notification. I agree with the proposed change, but the compatibility aspects of changes like this are always tricky to consider. In this case, the change is something that potentially weakens authorization. If a user has some automation that runs chmod on a directory, and that user expects the current behavior that sticky bit is preserved, then the effect would be to start allowing users to delete files owned by someone else. Admittedly, sticky bit usage is rare, typically only on /tmp, but I'd still be more comfortable with this as a 3.x change flagged backward-incompatible.
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks for providing the patch, Manoj Govindassamy

          • The changes on FsShellPermissions.java, UmaskParser.java and ChmodParser are not relevant? Lets remove them from the patch.
          • PermissionParser.java#73, is it a format only change?
          • In PermissionParser#applyOctalPattern:
            stickyBitType = userType = groupType = othersType = '=';
            

            Should we only set stickyBitType='='?

          • TestStickyBit.java. Please clean up imports.
            // Tear down the test directories
            402	    hdfs.delete(sbExplicitTestDir, true);
            403	    hdfs.delete(sbOmittedTestDir, true);
            404	    assertFalse(hdfs.exists(sbExplicitTestDir));
            405	    assertFalse(hdfs.exists(sbOmittedTestDir));
            

            You dont need to tear down these directories. Also you might not be necessary to test the sticky pit persistent by restart the cluster, as this is covered in the other test cases.

          The rest looks good. Thanks

          eddyxu Lei (Eddy) Xu added a comment - Thanks for providing the patch, Manoj Govindassamy The changes on FsShellPermissions.java , UmaskParser.java and ChmodParser are not relevant? Lets remove them from the patch. PermissionParser.java#73 , is it a format only change? In PermissionParser#applyOctalPattern : stickyBitType = userType = groupType = othersType = '=' ; Should we only set stickyBitType='=' ? TestStickyBit.java . Please clean up imports. // Tear down the test directories 402 hdfs.delete(sbExplicitTestDir, true ); 403 hdfs.delete(sbOmittedTestDir, true ); 404 assertFalse(hdfs.exists(sbExplicitTestDir)); 405 assertFalse(hdfs.exists(sbOmittedTestDir)); You dont need to tear down these directories. Also you might not be necessary to test the sticky pit persistent by restart the cluster, as this is covered in the other test cases. The rest looks good. Thanks

          The test failure was filed in HDFS-10696, and it was caused by HDFS-9937.

          weichiu Wei-Chiu Chuang added a comment - The test failure was filed in HDFS-10696 , and it was caused by HDFS-9937 .

          TestHDFSCLI failure is not related to the patch. Latest trunk is seeing the same failure.

          17699 2016-07-27 10:30:42,785 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(186)) -
          17700 2016-07-27 10:30:42,785 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(190)) - Summary results:
          17701 2016-07-27 10:30:42,785 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(191)) - ----------------------------------
          17702 
          17703 2016-07-27 10:30:42,785 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(211)) -                Testing mode: test
          17704 2016-07-27 10:30:42,785 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(212)) -
          17705 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(213)) -              Overall result: --- FAIL ---
          17706 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(221)) -                # Tests pass: 665 (99%)
          17707 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(223)) -                # Tests fail: 1 (0%)
          17708 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(227)) -          # Validations done: 1726 (each test may do mu      ltiple validations)
          17709 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(230)) -
          17710 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(231)) - Failing tests:
          17711 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(232)) - --------------17712 2016-07-27 10:30:42,786 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(238)) - 595: help: help for dfsadmin clrSpaceQuota
          17713 2016-07-27 10:30:42,787 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(248)) -17714 2016-07-27 10:30:42,787 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(249)) - Passing tests:
          17715 2016-07-27 10:30:42,787 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(250)) - --------------
          17716 2016-07-27 10:30:42,787 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(254)) - 1: ls: file using absolute path
          17717 2016-07-27 10:30:42,787 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(254)) - 2: ls: file using relative path
          
          
          
          17646 2016-07-27 10:45:22,083 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(177)) -                  Comparator: [RegexpComparator]
          17647 2016-07-27 10:45:22,083 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(179)) -          Comparision result:   [fail]
          17648 2016-07-27 10:45:22,083 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(181)) -             Expected output:   [^( |\t)*The storage ty      pe specific quota is cleared when -storageType option is specified.( )*]
          17649 2016-07-27 10:45:22,083 [main] INFO  cli.CLITestHelper (CLITestHelper.java:displayResults(183)) -               Actual output:   [-clrSpaceQuota [-stora      geType <storagetype>] <dirname>...<dirname>: Clear the space quota for each directory <dirName>.
          17650                 For each directory, attempt to clear the quota. An error will be reported if
          17651                 1. the directory does not exist or is a file, or
          17652                 2. user is not an administrator.
          17653                 It does not fault if the directory has no quota.
          17654                 The storage type specific quota is cleared when -storageType option is specified.               Available storageTypes are
          17655                 - RAM_DISK
          17656                 - DISK
          17657                 - SSD
          17658                 - ARCHIVE
          
          
          manojg Manoj Govindassamy added a comment - TestHDFSCLI failure is not related to the patch. Latest trunk is seeing the same failure. 17699 2016-07-27 10:30:42,785 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(186)) - 17700 2016-07-27 10:30:42,785 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(190)) - Summary results: 17701 2016-07-27 10:30:42,785 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(191)) - ---------------------------------- 17702 17703 2016-07-27 10:30:42,785 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(211)) - Testing mode: test 17704 2016-07-27 10:30:42,785 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(212)) - 17705 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(213)) - Overall result: --- FAIL --- 17706 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(221)) - # Tests pass: 665 (99%) 17707 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(223)) - # Tests fail: 1 (0%) 17708 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(227)) - # Validations done: 1726 (each test may do mu ltiple validations) 17709 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(230)) - 17710 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(231)) - Failing tests: 17711 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(232)) - --------------17712 2016-07-27 10:30:42,786 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(238)) - 595: help: help for dfsadmin clrSpaceQuota 17713 2016-07-27 10:30:42,787 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(248)) -17714 2016-07-27 10:30:42,787 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(249)) - Passing tests: 17715 2016-07-27 10:30:42,787 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(250)) - -------------- 17716 2016-07-27 10:30:42,787 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(254)) - 1: ls: file using absolute path 17717 2016-07-27 10:30:42,787 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(254)) - 2: ls: file using relative path 17646 2016-07-27 10:45:22,083 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(177)) - Comparator: [RegexpComparator] 17647 2016-07-27 10:45:22,083 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(179)) - Comparision result: [fail] 17648 2016-07-27 10:45:22,083 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(181)) - Expected output: [^( |\t)*The storage ty pe specific quota is cleared when -storageType option is specified.( )*] 17649 2016-07-27 10:45:22,083 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(183)) - Actual output: [-clrSpaceQuota [-stora geType <storagetype>] <dirname>...<dirname>: Clear the space quota for each directory <dirName>. 17650 For each directory, attempt to clear the quota. An error will be reported if 17651 1. the directory does not exist or is a file, or 17652 2. user is not an administrator. 17653 It does not fault if the directory has no quota. 17654 The storage type specific quota is cleared when -storageType option is specified. Available storageTypes are 17655 - RAM_DISK 17656 - DISK 17657 - SSD 17658 - ARCHIVE
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 6m 47s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 50s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 9s trunk passed
          +1 javadoc 1m 41s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 7m 4s the patch passed
          +1 javac 7m 4s the patch passed
          -0 checkstyle 1m 33s root: The patch generated 11 new + 268 unchanged - 2 fixed = 279 total (was 270)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 39s the patch passed
          +1 javadoc 1m 40s the patch passed
          +1 unit 7m 8s hadoop-common in the patch passed.
          -1 unit 58m 7s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          108m 39s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820332/HDFS-10689.001.patch
          JIRA Issue HDFS-10689
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8fc8ab6c6d25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d84ab8a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16202/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16202/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16202/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16202/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 50s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 41s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 7m 4s the patch passed +1 javac 7m 4s the patch passed -0 checkstyle 1m 33s root: The patch generated 11 new + 268 unchanged - 2 fixed = 279 total (was 270) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 39s the patch passed +1 javadoc 1m 40s the patch passed +1 unit 7m 8s hadoop-common in the patch passed. -1 unit 58m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 108m 39s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820332/HDFS-10689.001.patch JIRA Issue HDFS-10689 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8fc8ab6c6d25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d84ab8a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16202/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16202/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16202/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16202/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          manojg Manoj Govindassamy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          manojg Manoj Govindassamy made changes -
          Field Original Value New Value
          Attachment HDFS-10689.001.patch [ 12820332 ]

          Attaching a patch to make hdfs dfs chmod handle sticky bit permissions just like other filesystems.

          Test results with the proposed fix:

          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 1 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -mkdir /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod +t /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod -t /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod 0755 /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-t   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /test_dir
          manoj@~/work/hadev-pp: hdfs dfs -ls /
          Found 2 items
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:15 /test_dir
          drwxr-xr-x   - manoj supergroup          0 2016-07-26 15:13 /user
          
          manojg Manoj Govindassamy added a comment - Attaching a patch to make hdfs dfs chmod handle sticky bit permissions just like other filesystems. Test results with the proposed fix: manoj@~/work/hadev-pp: hdfs dfs -ls / Found 1 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -mkdir /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod +t /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-t - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod -t /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-t - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod 0755 /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod 1755 /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-t - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user manoj@~/work/hadev-pp: hdfs dfs -chmod 755 /test_dir manoj@~/work/hadev-pp: hdfs dfs -ls / Found 2 items drwxr-xr-x - manoj supergroup 0 2016-07-26 15:15 /test_dir drwxr-xr-x - manoj supergroup 0 2016-07-26 15:13 /user
          andrew.wang Andrew Wang added a comment -

          Thanks for digging into this Manoj. I'm of the opinion that we should try to imitate the Linux/BSD behavior, since that'll be least surprising to end users.

          Now the question is if we declare this a bug fix that can be backported to branch-2, or if this behavior change is too incompatible. Given that sticky bits are pretty rare in general, I think it's safe for branch-2, but would welcome other's thoughts. Anything to add Chris Nauroth?

          andrew.wang Andrew Wang added a comment - Thanks for digging into this Manoj. I'm of the opinion that we should try to imitate the Linux/BSD behavior, since that'll be least surprising to end users. Now the question is if we declare this a bug fix that can be backported to branch-2, or if this behavior change is too incompatible. Given that sticky bits are pretty rare in general, I think it's safe for branch-2, but would welcome other's thoughts. Anything to add Chris Nauroth ?

          Whenever the leading bit in the numeric format is omitted, it is considered to be all 0s. Chmod man page clearly mentions on the expected behavior with permission apply via numeric option with respect to setuid and setgid bits.

          man chmod:

          SETUID AND SETGID BITS
                 chmod clears the set-group-ID bit of a regular file if the file’s group ID does not match the user’s effective group ID or one of  the  user’s  supplementary  group
                 IDs,  unless  the  user  has  appropriate privileges.  Additional restrictions may cause the set-user-ID and set-group-ID bits of MODE or RFILE to be ignored.  This
                 behavior depends on the policy and functionality of the underlying chmod system call.  When in doubt, check the underlying system behavior.
          
                 chmod preserves a directory’s set-user-ID and set-group-ID bits unless you explicitly specify otherwise.  *You can set or clear the bits with symbolic modes like u+s
                 and g-s, and you can set (but not clear) the bits with a numeric mode.*
          

          That is, 755 will not reset setuid and setgid bits on the file/directory. However, man page doesn't mention anything of that sort for sticky bits. So, I am leaning towards how other distros have implemented this. Here is the EXT4 behavior.

          -bash-4.1$ df -T
          Filesystem     Type  1K-blocks      Used Available Use% Mounted on
          /dev/xvda1     ext4  103209948  91990624   5977880  94% /
          
          -bash-4.1$ pwd
          /home/manojg
          
          drwxrwxr-x 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit
          -bash-4.1$ chmod 1775 dir_test_sticky_bit
          -bash-4.1$ ls -l
          total 4
          drwxrwxr-t 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit
          
          -bash-4.1$ chmod 775 dir_test_sticky_bit
          -bash-4.1$ ls -l
          total 4
          drwxrwxr-x 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit  <=== 755 does clear out sticky bit
          -bash-4.1$ .
          

          So, EXT4 and many other filesystems on Linux, MacOSX are resetting Sticky Bit when the bit is not specified in the permission arg.

          manojg Manoj Govindassamy added a comment - Whenever the leading bit in the numeric format is omitted, it is considered to be all 0s. Chmod man page clearly mentions on the expected behavior with permission apply via numeric option with respect to setuid and setgid bits. man chmod: SETUID AND SETGID BITS chmod clears the set-group-ID bit of a regular file if the file’s group ID does not match the user’s effective group ID or one of the user’s supplementary group IDs, unless the user has appropriate privileges. Additional restrictions may cause the set-user-ID and set-group-ID bits of MODE or RFILE to be ignored. This behavior depends on the policy and functionality of the underlying chmod system call. When in doubt, check the underlying system behavior. chmod preserves a directory’s set-user-ID and set-group-ID bits unless you explicitly specify otherwise. *You can set or clear the bits with symbolic modes like u+s and g-s, and you can set (but not clear) the bits with a numeric mode.* That is, 755 will not reset setuid and setgid bits on the file/directory. However, man page doesn't mention anything of that sort for sticky bits. So, I am leaning towards how other distros have implemented this. Here is the EXT4 behavior. -bash-4.1$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/xvda1 ext4 103209948 91990624 5977880 94% / -bash-4.1$ pwd /home/manojg drwxrwxr-x 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit -bash-4.1$ chmod 1775 dir_test_sticky_bit -bash-4.1$ ls -l total 4 drwxrwxr-t 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit -bash-4.1$ chmod 775 dir_test_sticky_bit -bash-4.1$ ls -l total 4 drwxrwxr-x 2 manojg manojg 4096 Jul 25 19:09 dir_test_sticky_bit <=== 755 does clear out sticky bit -bash-4.1$ . So, EXT4 and many other filesystems on Linux, MacOSX are resetting Sticky Bit when the bit is not specified in the permission arg.
          manojg Manoj Govindassamy created issue -

          People

            manojg Manoj Govindassamy
            manojg Manoj Govindassamy
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack