Hadoop Common
  1. Hadoop Common
  2. HADOOP-8025

change default distcp log location to be /tmp rather than cwd

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Trivial Trivial
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.

        Activity

        Hide
        Allen Wittenauer added a comment -

        [I have no idea where 'tools' is supposed to go, so I'm going with common. Right now, this is horribly confusing.]

        Show
        Allen Wittenauer added a comment - [I have no idea where 'tools' is supposed to go, so I'm going with common. Right now, this is horribly confusing.]
        Hide
        Brian Bockelman added a comment -

        Hi Allen,

        Writing into /tmp is pretty scary to me due to things like symlink attacks. (If I want to find a security bug in a project, a simple place to start is to grep for usage of "/tmp"). It's not obvious to me whether or not the suggested patch opens distcp to symlink attacks.

        If the proper flags are indeed used later on, I would suggest adding a comment to this patch pointing out why its not a problem.

        Brian

        PS - I'm probably a touch grouchy about this because I file tickets for user applications approximately monthly for leaving cruft in /tmp! Once it gets in the code, it never seems to escape.

        Show
        Brian Bockelman added a comment - Hi Allen, Writing into /tmp is pretty scary to me due to things like symlink attacks. (If I want to find a security bug in a project, a simple place to start is to grep for usage of "/tmp"). It's not obvious to me whether or not the suggested patch opens distcp to symlink attacks. If the proper flags are indeed used later on, I would suggest adding a comment to this patch pointing out why its not a problem. Brian PS - I'm probably a touch grouchy about this because I file tickets for user applications approximately monthly for leaving cruft in /tmp! Once it gets in the code, it never seems to escape.
        Hide
        Allen Wittenauer added a comment -

        FWIW, we forcibly remove stuff from /tmp after 3 days via a simple perl script that does time conversions etc. Putting that policy in place + enforcement deals with the brokenness that is Pig and just general users being bad.

        But right now, instead of leaving this cruft in /tmp, distcp is likely spreading zero length files throughout your entire file system. Luckily, distcp uses a fairly easily recognizable and random filename. With this patch, this is now in /tmp and easily dealt a rm.

        Like most of my patches, I expect this patch has 0 chance of actually getting committed and is only here for others that want a fix.

        Show
        Allen Wittenauer added a comment - FWIW, we forcibly remove stuff from /tmp after 3 days via a simple perl script that does time conversions etc. Putting that policy in place + enforcement deals with the brokenness that is Pig and just general users being bad. But right now, instead of leaving this cruft in /tmp, distcp is likely spreading zero length files throughout your entire file system. Luckily, distcp uses a fairly easily recognizable and random filename. With this patch, this is now in /tmp and easily dealt a rm. Like most of my patches, I expect this patch has 0 chance of actually getting committed and is only here for others that want a fix.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/)
        HADOOP-8025. Amendment (Revision 1297276)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/ ) HADOOP-8025 . Amendment (Revision 1297276) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276 Files : /hadoop/common/trunk/hadoop-client/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/)
        HADOOP-8025. Amendment (Revision 1297276)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/ ) HADOOP-8025 . Amendment (Revision 1297276) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276 Files : /hadoop/common/trunk/hadoop-client/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/)
        HADOOP-8025. Amendment (Revision 1297276)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/ ) HADOOP-8025 . Amendment (Revision 1297276) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276 Files : /hadoop/common/trunk/hadoop-client/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/)
        HADOOP-8025. Amendment (Revision 1297276)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/ ) HADOOP-8025 . Amendment (Revision 1297276) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276 Files : /hadoop/common/trunk/hadoop-client/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/)
        HADOOP-8025. Amendment (Revision 1297276)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/ ) HADOOP-8025 . Amendment (Revision 1297276) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276 Files : /hadoop/common/trunk/hadoop-client/pom.xml
        Hide
        Allen Wittenauer added a comment -

        I think someone has the wrong jira tagged in their commit or Jenkins has gone insane.

        Show
        Allen Wittenauer added a comment - I think someone has the wrong jira tagged in their commit or Jenkins has gone insane.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        It seems that the commit was for HADOOP-8125.

        ------------------------------------------------------------------------
        r1297276 | tucu | 2012-03-05 15:10:04 -0800 (Mon, 05 Mar 2012) | 1 line
        
        HADOOP-8025. Amendment
        ------------------------------------------------------------------------
        ...
        
        ------------------------------------------------------------------------
        r1297270 | tucu | 2012-03-05 15:04:19 -0800 (Mon, 05 Mar 2012) | 1 line
        
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu)
        
        Show
        Tsz Wo Nicholas Sze added a comment - It seems that the commit was for HADOOP-8125 . ------------------------------------------------------------------------ r1297276 | tucu | 2012-03-05 15:10:04 -0800 (Mon, 05 Mar 2012) | 1 line HADOOP-8025. Amendment ------------------------------------------------------------------------ ... ------------------------------------------------------------------------ r1297270 | tucu | 2012-03-05 15:04:19 -0800 (Mon, 05 Mar 2012) | 1 line HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu)

          People

          • Assignee:
            Unassigned
            Reporter:
            Allen Wittenauer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development