Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5022

[HOD] logcondense should delete all hod logs for a user, including jobtracker logs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New logcondense option retain-master-logs indicates whether the script should delete master logs as part of its cleanup process. By default this option is false; master logs are deleted. Earlier versions of logcondense did not delete master logs.

      Description

      Currently, logcondense.py does not delete jobtracker logs that it uploads to the DFS when the HOD cluster is deallocated. This will result in the hod-logs directory to slowly accumulate a whole bunch of jobtracker logs. Particularly for users who run a lot of user jobs, this could fill up the namespace. Further these directories will cause the logcondense program to keep repeatedly looking at these directories stressing out the namenode. So, logcondense.py should optionally also delete the jobtracker logs.

        Attachments

        1. hadoop-5022.txt
          3 kB
          Peeyush Bishnoi
        2. hadoop-5022-1.txt
          4 kB
          Peeyush Bishnoi
        3. hadoop-5022-2.txt
          4 kB
          Peeyush Bishnoi
        4. hadoop-5022-3.txt
          4 kB
          Hemanth Yamijala

          Activity

            People

            • Assignee:
              peeyushb Peeyush Bishnoi
              Reporter:
              yhemanth Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: