Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10945

4-digit octal umask permissions throws a parse error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Providing a 4-digit octal number for fs permissions leads to a parse error, e.g.: -Dfs.permissions.umask-mode=0022

      1. HADOOP-10945.2.patch
        2 kB
        Chang Li
      2. HADOOP-10945.3.patch
        3 kB
        Chang Li
      3. HADOOP-10945.patch
        2 kB
        Chang Li

        Issue Links

          Activity

          Hide
          jlowe Jason Lowe added a comment -

          Example failure:

          $ hadoop fs -Dfs.permissions.umask-mode=0022 -ls
          2014-08-07 16:10:27,811 WARN  [main] permission.FsPermission (FsPermission.java:getUMask(260)) - Unable to parse configuration fs.permissions.umask-mode with value 0022 as octal or symbolic umask.
          -ls: Unable to parse configuration fs.permissions.umask-mode with value 0022 as octal or symbolic umask.
          Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...]
          

          The pattern match for a valid octal pattern in UmaskParser probably needs to look a little more like the one from ChmodParser. I realize that the umask can't specify the sticky bit, but we shouldn't preclude an equivalent number to a valid umask.

          Show
          jlowe Jason Lowe added a comment - Example failure: $ hadoop fs -Dfs.permissions.umask-mode=0022 -ls 2014-08-07 16:10:27,811 WARN [main] permission.FsPermission (FsPermission.java:getUMask(260)) - Unable to parse configuration fs.permissions.umask-mode with value 0022 as octal or symbolic umask. -ls: Unable to parse configuration fs.permissions.umask-mode with value 0022 as octal or symbolic umask. Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...] The pattern match for a valid octal pattern in UmaskParser probably needs to look a little more like the one from ChmodParser. I realize that the umask can't specify the sticky bit, but we shouldn't preclude an equivalent number to a valid umask.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 4, now 4).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 11s Tests passed in hadoop-common.
          -1 hdfs tests 161m 8s Tests failed in hadoop-hdfs.
              228m 3s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747252/HADOOP-10945.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 156f24e
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 4, now 4). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 11s Tests passed in hadoop-common. -1 hdfs tests 161m 8s Tests failed in hadoop-hdfs.     228m 3s   Reason Tests Failed unit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747252/HADOOP-10945.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 156f24e checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7340/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Chang Li for taking this issue.

          -    Pattern.compile("^\\s*[+]?()([0-7]{3})\\s*$"); // no leading 1 for sticky bit
          +    Pattern.compile("^\\s*[+]?([0]*)([0-7]{3})\\s*$"); // no leading 1 for sticky bit
          

          [0]* can pass an argument which is more than 4-digit (ex. 00022 or 000022). Would you use 0? instead?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Chang Li for taking this issue. - Pattern.compile( "^\\s*[+]?()([0-7]{3})\\s*$" ); // no leading 1 for sticky bit + Pattern.compile( "^\\s*[+]?([0]*)([0-7]{3})\\s*$" ); // no leading 1 for sticky bit [0]* can pass an argument which is more than 4-digit (ex. 00022 or 000022). Would you use 0? instead?
          Hide
          lichangleo Chang Li added a comment -

          Akira Ajisaka thanks for review. I think even umask 00022 or 000022 should not fail the parser and it will be nice for parser to translate them into a valid umask. Jason Lowe could you comment on this?

          Show
          lichangleo Chang Li added a comment - Akira Ajisaka thanks for review. I think even umask 00022 or 000022 should not fail the parser and it will be nice for parser to translate them into a valid umask. Jason Lowe could you comment on this?
          Hide
          jlowe Jason Lowe added a comment -

          The umask is taking an octal number, and IMHO it should accept any legal octal number that fits in the expected range of values. I do not understand at all why the parsing code isn't just calling Integer.parseInt to parse the octal number and extract each three-bit value from that. The regexp seems like a very heavy-handed and error-prone way to do it, but I don't know if we need to rewrite the parsing from the ground up just to fix this particular error.

          Patch looks OK overall. Could you add some tests for invalid permissions as well? e.g.: 1222 and 01222 are not supposed to be valid.

          Show
          jlowe Jason Lowe added a comment - The umask is taking an octal number, and IMHO it should accept any legal octal number that fits in the expected range of values. I do not understand at all why the parsing code isn't just calling Integer.parseInt to parse the octal number and extract each three-bit value from that. The regexp seems like a very heavy-handed and error-prone way to do it, but I don't know if we need to rewrite the parsing from the ground up just to fix this particular error. Patch looks OK overall. Could you add some tests for invalid permissions as well? e.g.: 1222 and 01222 are not supposed to be valid.
          Hide
          lichangleo Chang Li added a comment -

          Jason Lowe thanks for review! added tests for invalid umask.

          Show
          lichangleo Chang Li added a comment - Jason Lowe thanks for review! added tests for invalid umask.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 53s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 0s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 15s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 10s Tests passed in hadoop-common.
          +1 hdfs tests 159m 19s Tests passed in hadoop-hdfs.
              225m 9s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747371/HADOOP-10945.3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 2196e39
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 53s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 10s Tests passed in hadoop-common. +1 hdfs tests 159m 19s Tests passed in hadoop-hdfs.     225m 9s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747371/HADOOP-10945.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2196e39 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7347/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1

          Show
          ajisakaa Akira Ajisaka added a comment - +1
          Hide
          jlowe Jason Lowe added a comment -

          +1, committing this.

          Show
          jlowe Jason Lowe added a comment - +1, committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Chang for the contribution and to Akira for additional review! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks to Chang for the contribution and to Akira for additional review! I committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8240 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8240/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8240 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8240/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1002 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1002/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1002 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1002/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #272 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/272/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #272 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/272/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/261/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #261 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/261/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2199/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2199/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #269 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/269/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #269 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/269/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2218/)
          HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2218 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2218/ ) HADOOP-10945 . 4-digit octal umask permissions throws a parse error. Contributed by Chang Li (jlowe: rev 6f0a35724f0da80146dbae4b6f6c341e1d3101f5) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/UmaskParser.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java

            People

            • Assignee:
              lichangleo Chang Li
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development