Hadoop Common
  1. Hadoop Common
  2. HADOOP-5113

logcondense should delete hod logs for a user , whose username has any of the characters in the value passed to "-l" options

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Logcondense script is not able to delete the completed job directories of hadoop logs in hod-logs inside HDFS for the the users , whose username has any of the characters , in the value passed to "-l" options or in '/user' as set default . This happened because logcondense script use python 'lstrip' method , which returns copy of the string after removing leading characters in the value passed to "-l" options or in "/user" instead of just stripping value from the given string .

      1. hadoop-5113-0.18.txt
        0.6 kB
        Robert Chansler
      2. hadoop-5113.txt
        0.6 kB
        Peeyush Bishnoi

        Activity

        Hide
        Peeyush Bishnoi added a comment -

        Example for 'lstrip' illustration :

        Suppose there is file on HDFS :

        > filename='/user/shum/test.txt'

        we need to strip off the '/user' from filename ,

        > filename=filename.lstrip('/user')

        if we print the above statement , the output would be :

        > hum/test.txt

        instead of

        > shum/test.txt

        here lstrip method has strip off value and produce the incorrect output .

        So we need to use len method for correct strip off the value.

        > filename=filename[len('/user'):]

        output of above statement would be :

        > /shum/test.txt

        Show
        Peeyush Bishnoi added a comment - Example for 'lstrip' illustration : Suppose there is file on HDFS : > filename='/user/shum/test.txt' we need to strip off the '/user' from filename , > filename=filename.lstrip('/user') if we print the above statement , the output would be : > hum/test.txt instead of > shum/test.txt here lstrip method has strip off value and produce the incorrect output . So we need to use len method for correct strip off the value. > filename=filename [len('/user'):] output of above statement would be : > /shum/test.txt —
        Hide
        Peeyush Bishnoi added a comment -

        Patch is attached that will remove the lstrip method and use len method on value passed to '-l' options or on '/user' (set as default ).

        Comments ?

        Show
        Peeyush Bishnoi added a comment - Patch is attached that will remove the lstrip method and use len method on value passed to '-l' options or on '/user' (set as default ). Comments ? —
        Hide
        Hemanth Yamijala added a comment -

        Sigh ! (for the issue)

        +1 for the fix.

        Show
        Hemanth Yamijala added a comment - Sigh ! (for the issue) +1 for the fix.
        Hide
        Hemanth Yamijala added a comment -

        test-patch results:

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no tests are needed for this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        HOD tests are run manually. And this test has been independently verified.

        Show
        Hemanth Yamijala added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. HOD tests are run manually. And this test has been independently verified.
        Hide
        Peeyush Bishnoi added a comment -

        The patch has been successfully applied on Hadoop 0.18 , 0.19 and 0.20 branches .

        Show
        Peeyush Bishnoi added a comment - The patch has been successfully applied on Hadoop 0.18 , 0.19 and 0.20 branches . —
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Peeyush !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Peeyush !
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #756 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/ )
        Hide
        Robert Chansler added a comment -

        Example for earlier versions, not to be committed.

        Show
        Robert Chansler added a comment - Example for earlier versions, not to be committed.

          People

          • Assignee:
            Peeyush Bishnoi
            Reporter:
            Peeyush Bishnoi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development