Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33215

Speed up event log download by skipping UI rebuild

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.7, 3.0.1
    • Fix Version/s: 3.1.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      Right now, when we want to download the event logs from the spark history server(SHS), SHS will need to parse entire the event log to rebuild UI, and this is just for view permission checks. UI rebuilding is a time-consuming and memory-intensive task, especially for large logs. However, this process is unnecessary for event log download.

      This patch enables SHS to check UI view permissions of a given app/attempt for a given user, without rebuilding the UI. This is achieved by adding a method "checkUIViewPermissions(appId: String, attemptId: Option[String], user: String): Boolean" to many layers of history server components.

      With this patch, UI rebuild can be skipped when downloading event logs from the history server. Thus the time of downloading a GB scale event log can be reduced from several minutes to several seconds, and the memory consumption of UI rebuilding can be avoided.

        Attachments

          Activity

            People

            • Assignee:
              Baohe Zhang Baohe Zhang
              Reporter:
              Baohe Zhang Baohe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: