Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17003

No Log compression and retention in Hadoop

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: kms
    • Labels:
      None

      Description

      Hadoop logging lacks several important features. Logs generated end up eating disk space
      We need an implementation that satisfies the following three features: 1) time-based rolling, 2) retention and 3) compression.

      For example KMS logs have no retention or compression.

      -rw-r--r-- 1 hkms users 704M Mar 20 23:59 kms.log.2020-03-20
      -rw-r--r-- 1 hkms users 731M Mar 21 23:59 kms.log.2020-03-21
      -rw-r--r-- 1 hkms users 750M Mar 22 23:59 kms.log.2020-03-22
      -rw-r--r-- 1 hkms users 757M Mar 23 23:59 kms.log.2020-03-23
      -rw-r--r-- 1 hkms users 805M Mar 24 23:59 kms.log.2020-03-24
      -rw-r--r-- 1 hkms users 858M Mar 25 23:59 kms.log.2020-03-25
      -rw-r--r-- 1 hkms users 875M Mar 26 23:59 kms.log.2020-03-26
      -rw-r--r-- 1 hkms users 754M Mar 27 23:59 kms.log.2020-03-27
      

        Attachments

        1. HADOOP-17003.001.patch
          31 kB
          Ahmed Hussein
        2. HADOOP-17003.002.patch
          32 kB
          Ahmed Hussein
        3. HADOOP-17003.003.patch
          33 kB
          Ahmed Hussein
        4. HADOOP-17003.004.patch
          33 kB
          Ahmed Hussein
        5. HADOOP-17003.005.patch
          33 kB
          Ahmed Hussein
        6. HADOOP-17003.006.patch
          33 kB
          Ahmed Hussein

          Issue Links

            Activity

              People

              • Assignee:
                ahussein Ahmed Hussein
                Reporter:
                ahussein Ahmed Hussein
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: