Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20084

Remove internal.metrics.updatedBlockStatuses accumulator from history files

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.2, 2.2.0
    • Component/s: Spark Core, Web UI
    • Labels:
      None

      Description

      History files for large jobs can be hundreds of GB. These history files take too much space and create a backlog on the history server.

      Most of the size is from Accumulables in SparkListenerTaskEnd. The largest accumulable is internal.metrics.updatedBlockStatuses, which has a small update (the blocks that were changed) but a huge value (all known blocks). Nothing currently uses the accumulator value or update, so it is safe to remove it. Information for any block updated during a task is also recorded under Task Metrics / Updated Blocks.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rdblue Ryan Blue
                Reporter:
                rdblue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: