Hadoop Common
  1. Hadoop Common
  2. HADOOP-6702

Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.19.1, 0.20.0, 0.20.1, 0.20.2
    • Fix Version/s: 0.20.3, 0.21.0, 0.22.0
    • Component/s: fs
    • Labels:
      None

      Description

      Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.

      This bug is due to missing update of errors count in FsShell.java.

      FsShell.java
      int runCmdHandler(CmdHandler handler, String[] args,
                                         int startIndex, boolean recursive) 
                                         throws IOException {
          int errors = 0;
          
          for (int i=startIndex; i<args.length; i++) {
            Path srcPath = new Path(args[i]);
            FileSystem srcFs = srcPath.getFileSystem(getConf());
            Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
            for(Path path : paths) {
              try {
                FileStatus file = srcFs.getFileStatus(path);
                if (file == null) {
                  System.err.println(handler.getName() + 
                                     ": could not get status for '" + path + "'");
                  errors++;
                } else {
                  errors += runCmdHandler(handler, file, srcFs, recursive);
                }
              } catch (IOException e) {
                String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
                  (e.getCause().getMessage() != null ? 
                      e.getCause().getLocalizedMessage() : "null"));
                System.err.println(handler.getName() + ": could not get status for '"
                                              + path + "': " + msg.split("\n")[0]);
                errors++;
              }
            }
          }
       

      If there are no files on HDFS matching to wildcard input then srcFs.globStatus(srcpath) returns 0.
      {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}

      Resulting no increment in errors and command exits with 0 even though file/directory does not exist.

        Issue Links

          Activity

          Hide
          Ravi Phulari added a comment -

          Should I create a separate Jira for glob handling?

          Koji, could you please create Jira and explain desired fix for this error.

          Show
          Ravi Phulari added a comment - Should I create a separate Jira for glob handling? Koji, could you please create Jira and explain desired fix for this error.
          Hide
          Koji Noguchi added a comment -

          HADOOP-6701 Fixes this issue.

          Ravi, HADOOP-6701 just moved the error handling of globbing from one type to another.
          Should I create a separate Jira for glob handling?

          Show
          Koji Noguchi added a comment - HADOOP-6701 Fixes this issue. Ravi, HADOOP-6701 just moved the error handling of globbing from one type to another. Should I create a separate Jira for glob handling?
          Hide
          Ravi Phulari added a comment -

          HADOOP-6701 Fixes this issue.

          Show
          Ravi Phulari added a comment - HADOOP-6701 Fixes this issue.
          Hide
          Koji Noguchi added a comment -

          Incorrect return code for wildcard in hadoop is not limited to chown/chgrp.
          It's everywhere.
          For example,
          In 'ls', this is how unix performs,

          % ls nonexist*
          ls: No match.
          % echo $?
          1
          % ls nonexist* file*
          fileA
          % echo $?
          0
          % ls file* nonexist* 
          fileA
          % echo $?
          0
          

          It returns 0 as long as one of the globing matches.
          and in hadoop 'ls'

          % hadoop dfs -ls file\* nonexist\*
          Found 1 items
          -rw-------   3 knoguchi users          7 2010-04-08 15:57 /user/knoguchi/fileA
          ls: Cannot access nonexist*: No such file or directory.
          % echo $?
          255
          
          % hadoop dfs -ls nonexist\* file\* 
          ls: Cannot access nonexist*: No such file or directory.
          Found 1 items
          -rw-------   3 knoguchi users          7 2010-04-08 15:57 /user/knoguchi/fileA
          % echo $?
          0
          % 
          

          hadoop 'ls' simply returns the result of the last globbing.

          This behavior is also inconsistent from command to command.
          Picking three hadoop commands. chgrp/ls/du.

          command single globbing multiple globbing
          chown/chgrp/etc X (returns 0 even if globing returns empty) X (returns 0 even if all the globbing returns empty)
          ls O X (returns last globbing result)
          du O X (returns non-zero even if one of the globbing fail)

          Suggested fix in this Jira would simply change the behavior of 'chown/chgrp' to 'du'-like which means multiple globbing will still be incorrect.

          Show
          Koji Noguchi added a comment - Incorrect return code for wildcard in hadoop is not limited to chown/chgrp. It's everywhere. For example, In 'ls', this is how unix performs, % ls nonexist* ls: No match. % echo $? 1 % ls nonexist* file* fileA % echo $? 0 % ls file* nonexist* fileA % echo $? 0 It returns 0 as long as one of the globing matches . and in hadoop 'ls' % hadoop dfs -ls file\* nonexist\* Found 1 items -rw------- 3 knoguchi users 7 2010-04-08 15:57 /user/knoguchi/fileA ls: Cannot access nonexist*: No such file or directory. % echo $? 255 % hadoop dfs -ls nonexist\* file\* ls: Cannot access nonexist*: No such file or directory. Found 1 items -rw------- 3 knoguchi users 7 2010-04-08 15:57 /user/knoguchi/fileA % echo $? 0 % hadoop 'ls' simply returns the result of the last globbing. This behavior is also inconsistent from command to command. Picking three hadoop commands. chgrp/ls/du. command single globbing multiple globbing chown/chgrp/etc X (returns 0 even if globing returns empty) X (returns 0 even if all the globbing returns empty) ls O X (returns last globbing result) du O X (returns non-zero even if one of the globbing fail) Suggested fix in this Jira would simply change the behavior of 'chown/chgrp' to 'du'-like which means multiple globbing will still be incorrect.
          Hide
          Ravi Phulari added a comment -

          This issue fixes wildcard bug for incorrect exit code.

          Show
          Ravi Phulari added a comment - This issue fixes wildcard bug for incorrect exit code.

            People

            • Assignee:
              Ravi Phulari
              Reporter:
              Ravi Phulari
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development