Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17003

No Log compression and retention in Hadoop

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • kms
    • None

    Description

      Hadoop logging lacks several important features. Logs generated end up eating disk space
      We need an implementation that satisfies the following three features: 1) time-based rolling, 2) retention and 3) compression.

      For example KMS logs have no retention or compression.

      -rw-r--r-- 1 hkms users 704M Mar 20 23:59 kms.log.2020-03-20
      -rw-r--r-- 1 hkms users 731M Mar 21 23:59 kms.log.2020-03-21
      -rw-r--r-- 1 hkms users 750M Mar 22 23:59 kms.log.2020-03-22
      -rw-r--r-- 1 hkms users 757M Mar 23 23:59 kms.log.2020-03-23
      -rw-r--r-- 1 hkms users 805M Mar 24 23:59 kms.log.2020-03-24
      -rw-r--r-- 1 hkms users 858M Mar 25 23:59 kms.log.2020-03-25
      -rw-r--r-- 1 hkms users 875M Mar 26 23:59 kms.log.2020-03-26
      -rw-r--r-- 1 hkms users 754M Mar 27 23:59 kms.log.2020-03-27
      

      Attachments

        1. HADOOP-17003.006.patch
          33 kB
          Ahmed Hussein
        2. HADOOP-17003.005.patch
          33 kB
          Ahmed Hussein
        3. HADOOP-17003.004.patch
          33 kB
          Ahmed Hussein
        4. HADOOP-17003.003.patch
          33 kB
          Ahmed Hussein
        5. HADOOP-17003.002.patch
          32 kB
          Ahmed Hussein
        6. HADOOP-17003.001.patch
          31 kB
          Ahmed Hussein

        Issue Links

          Activity

            People

              ahussein Ahmed Hussein
              ahussein Ahmed Hussein
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: