Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-760

"fs -put" fails if dfs.umask is set to 63

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      Add the following to hdfs-site.conf

        <property>
          <name>dfs.umask</name>
          <value>63</value>
        </property>
      

      Then run "hadoop fs -put"

      -bash-3.1$ ./bin/hadoop fs -put README.txt r.txt
      09/11/09 23:09:07 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
      put: 63
      Usage: java FsShell [-put <localsrc> ... <dst>]
      -bash-3.1$
      

      Observed the above behavior in 0.21.

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          Some info: This is being caused by a problem in the configuration. The stack trace:

          java.lang.IllegalArgumentException: 63
                  at org.apache.hadoop.fs.permission.PermissionParser.<init>(PermissionParser.java:54)
                  at org.apache.hadoop.fs.permission.UmaskParser.<init>(UmaskParser.java:37)
                  at org.apache.hadoop.fs.permission.FsPermission.getUMask(FsPermission.java:204)
                  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:569)
                  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:537)
                  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:213)
                  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:547)
                  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:528)
                  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
                  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:219)
                  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
                  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:156)
                  at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1533)
                  at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:129)
                  at org.apache.hadoop.fs.FsShell.run(FsShell.java:1837)
                  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
                  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
                  at org.apache.hadoop.fs.FsShell.main(FsShell.java:1974)
          

          is misleading.
          dfs.umask is the old-style key for setting the umask and is correctly read in by the Configuration as the old-style. However, the section of the code that determines whether or not to treat it as an old-style, decimal value (that should be converted to octal before being passed to the PermissionParser) is being given the wrong answer by the configuration:

              if(conf != null) {
                String confUmask = conf.get(UMASK_LABEL);
                if(confUmask != null)  // UMASK_LABEL is set
                  if(conf.deprecatedKeyWasSet(DEPRECATED_UMASK_LABEL)) { // <--- this is returning false but should be true
                    umask = Integer.parseInt(confUmask); // Evaluate as decimal value
                  else
                    umask = new UmaskParser(confUmask).getUMask();  // <----- therefore this tries to parse the decimal as padded octal and fails
          

          The code in Configuration that checks whether or not a deprecated value was set is returning false, though it shouldn't be (specifically deprecatedKeyMap.get(oldKey).accessed is still set to false and should be true). I'll look more tomorrow.

          Regardless of the reason, we should probably have better exception handling in the FsShell. The exception thrown from PermissionParser should be more descriptive so that when it hits the user there is a better sense of what went wrong.

          Show
          Jakob Homan added a comment - Some info: This is being caused by a problem in the configuration. The stack trace: java.lang.IllegalArgumentException: 63 at org.apache.hadoop.fs.permission.PermissionParser.<init>(PermissionParser.java:54) at org.apache.hadoop.fs.permission.UmaskParser.<init>(UmaskParser.java:37) at org.apache.hadoop.fs.permission.FsPermission.getUMask(FsPermission.java:204) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:569) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:537) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:213) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:547) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:528) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:219) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:156) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1533) at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:129) at org.apache.hadoop.fs.FsShell.run(FsShell.java:1837) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.FsShell.main(FsShell.java:1974) is misleading. dfs.umask is the old-style key for setting the umask and is correctly read in by the Configuration as the old-style. However, the section of the code that determines whether or not to treat it as an old-style, decimal value (that should be converted to octal before being passed to the PermissionParser) is being given the wrong answer by the configuration: if (conf != null ) { String confUmask = conf.get(UMASK_LABEL); if (confUmask != null ) // UMASK_LABEL is set if (conf.deprecatedKeyWasSet(DEPRECATED_UMASK_LABEL)) { // <--- this is returning false but should be true umask = Integer .parseInt(confUmask); // Evaluate as decimal value else umask = new UmaskParser(confUmask).getUMask(); // <----- therefore this tries to parse the decimal as padded octal and fails The code in Configuration that checks whether or not a deprecated value was set is returning false, though it shouldn't be (specifically deprecatedKeyMap.get(oldKey).accessed is still set to false and should be true). I'll look more tomorrow. Regardless of the reason, we should probably have better exception handling in the FsShell. The exception thrown from PermissionParser should be more descriptive so that when it hits the user there is a better sense of what went wrong.
          Hide
          Tom White added a comment -

          Downgrading from blocker for 0.21. Looks like this is a corner case which has a workaround (use octal).

          Show
          Tom White added a comment - Downgrading from blocker for 0.21. Looks like this is a corner case which has a workaround (use octal).
          Hide
          Jakob Homan added a comment -

          This was fixed by HADOOP-6521.

          Show
          Jakob Homan added a comment - This was fixed by HADOOP-6521 .

            People

            • Assignee:
              Unassigned
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development