Hadoop Common
  1. Hadoop Common
  2. HADOOP-9193

hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha, 0.23.5
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.7
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The hadoop front-end script will print a deprecation warning and defer to the hdfs front-end script for certain commands, like fsck, dfs. If a wildcard appears as an argument then it can be inadvertently expanded by the shell to match a local filesystem path before being sent to the hdfs script, which can be very confusing to the end user.

      For example, the following two commands usually perform very different things, even though they should be equivalent:

      hadoop fs -ls /tmp/\*
      hadoop dfs -ls /tmp/\*
      

      The former lists everything in the default filesystem under /tmp, while the latter expands /tmp/* into everything in the local filesystem under /tmp and passes those as arguments to try to list in the default filesystem.

      1. hadoop9193.diff
        1 kB
        Andy Isaacson

        Activity

        Hide
        Jason Lowe added a comment -

        Note that this may be somewhat moot if HDFS-2745 is committed and no other delegated commands are expected to take wildcard arguments or other types of arguments that could be accidentally interpreted by the shell during the exec.

        Show
        Jason Lowe added a comment - Note that this may be somewhat moot if HDFS-2745 is committed and no other delegated commands are expected to take wildcard arguments or other types of arguments that could be accidentally interpreted by the shell during the exec.
        Hide
        Andy Isaacson added a comment -

        The wrapper script is clearly wrong using $* , unquoted even. The correct construct is "$@". Attaching patch.

        Show
        Andy Isaacson added a comment - The wrapper script is clearly wrong using $* , unquoted even. The correct construct is "$@" . Attaching patch.
        Hide
        Andy Isaacson added a comment -

        Oops, sorry Robert Parker, I didn't mean to steal this jira from you (my display didn't even indicate that you'd assigned it). Feel free to steal back if you want.

        Show
        Andy Isaacson added a comment - Oops, sorry Robert Parker , I didn't mean to steal this jira from you (my display didn't even indicate that you'd assigned it). Feel free to steal back if you want.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12564049/hadoop9193.diff
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2019//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2019//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564049/hadoop9193.diff against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2019//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2019//console This message is automatically generated.
        Hide
        Robert Parker added a comment -

        Andy Isaacson No worries

        Show
        Robert Parker added a comment - Andy Isaacson No worries
        Hide
        Andy Isaacson added a comment -

        The TestZKFailoverController failure is unrelated. There does not seem to be any existing test code for the hadoop dfs shell scripts, so adding tests for this condition is challenging.

        Show
        Andy Isaacson added a comment - The TestZKFailoverController failure is unrelated. There does not seem to be any existing test code for the hadoop dfs shell scripts, so adding tests for this condition is challenging.
        Hide
        Colin Patrick McCabe added a comment -

        Seems like a good fix. Another layer of shell expansion doesn't help anyone here.

        Show
        Colin Patrick McCabe added a comment - Seems like a good fix. Another layer of shell expansion doesn't help anyone here.
        Hide
        Todd Lipcon added a comment -

        +1, lgtm. Will commit momentarily.

        Show
        Todd Lipcon added a comment - +1, lgtm. Will commit momentarily.
        Hide
        Aaron T. Myers added a comment -

        +1, the patch looks good to me.

        I'm going to commit this momentarily.

        Show
        Aaron T. Myers added a comment - +1, the patch looks good to me. I'm going to commit this momentarily.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3252 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3252/)
        HADOOP-9193. hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3252 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3252/ ) HADOOP-9193 . hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #99 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/99/)
        HADOOP-9193. hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #99 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/99/ ) HADOOP-9193 . hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1288 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1288/)
        HADOOP-9193. hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1288 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1288/ ) HADOOP-9193 . hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1316 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1316/)
        HADOOP-9193. hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1316 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1316/ ) HADOOP-9193 . hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. Contributed by Andy Isaacson. (Revision 1434450) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1434450 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #513 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/513/)
        HADOOP-9193. hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. (Andy Isaacson via tgraves) (Revision 1441499)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1441499
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #513 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/513/ ) HADOOP-9193 . hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script. (Andy Isaacson via tgraves) (Revision 1441499) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1441499 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop

          People

          • Assignee:
            Andy Isaacson
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development