Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17711

Compress rolled executor logs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.2, 2.2.0
    • None
    • None

    Description

      Currently, rolled executor logs are not compressed. If the executor produces a lot of logs, it may consume all executor disk space and fail the task. With this feature, executor will compress the rolled stderr/stdout like log4j to reduce disk usage.

      mengxr

      Attachments

        Activity

          People

            loneknightpy@gmail.com Yu Peng
            loneknightpy@gmail.com Yu Peng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: